On-device personalization of speech synthesis for training of speech model(s)

ABSTRACT

Processor(s) of a client device can: identify a textual segment stored locally at the client device; process the textual segment, using an on-device TTS generator model, to generate synthesized speech audio data that includes synthesized speech of the textual segment; process the synthesized speech, using an on-device ASR model to generate predicted ASR output; and generate a gradient based on comparing the predicted ASR output to ground truth output corresponding to the textual segment. Processor(s) of the client device can also: process the synthesized speech audio data using an on-device TTS generator model to make a prediction; and generate a gradient based on the prediction. In these implementations, the generated gradient(s) can be used to update weight(s) of the respective on-device model(s) and/or transmitted to a remote system for use in remote updating of respective global model(s). The updated weight(s) and/or the updated model(s) can be transmitted to client device(s).

BACKGROUND

Voice-based user interfaces are increasingly being used in the controlof computers and other electronic devices. Voice-based user interfaceshave continued to evolve from early rudimentary interfaces that couldonly understand simple and direct commands to more sophisticatedinterfaces that respond to natural language requests and that canunderstand context and manage back-and-forth dialogs or conversationswith users. Many voice-based user interfaces perform speech recognition(e.g., using a speech-to-text (STT) model) on spoken utterances togenerate corresponding text, perform a semantic analysis of thecorresponding text in an attempt to determine the meaning of the spokenutterances, undertake one or more actions based on the determinedmeaning, track a flow of each of the conversations, and annotate thespoken utterances and/or the corresponding text with an indication ofthe user that provided the spoken utterances. Some voice-based userinterfaces can also synthesize speech (e.g., using a text-to-speechmodel) based on text to generate corresponding synthesized speech audiodata, and audibly render the synthesized speech audio data at a clientdevice.

While performance of speech recognition has continued to improve,inaccurate speech recognition can still occur for many situations. As anon-limiting example, inaccurate speech recognition can occur for newterms and/or for terms that are relatively infrequent (or non-existent)in a training corpus on which a speech recognition model is trained. Inan attempt to effectively recognize new terms and/or infrequent terms,techniques have been proposed to generate additional speech recognitionhypotheses that are in addition to an initial hypothesis (or initialhypotheses), and consider the additional speech recognition hypothesesas candidates for speech recognition. However, such techniques requireadditional post-processing, and can still fail to lead to effectiverecognition of many terms in many situations, such as when the initialhypothesis/hypotheses are too far off-base and/or when a lexicon for theadditional hypotheses does not include certain terms.

Moreover, inaccurate speech recognition can be exacerbated when speechrecognition is performed on-device (i.e., on a client device). This canbe due to, for example, an on-device speech recognition model being lessrobust than a cloud-based model, on-device memory and/or processorresources being more constrained than cloud-based resources, and/oradditional hypotheses generation lexicons or language models being moreconstrained on device.

SUMMARY

Some implementations disclosed herein are directed to improvingperformance of speech recognition that is performed locally, at a clientdevice, utilizing an on-device automatic speech recognition (ASR) modelstored locally at the client device. In some of those implementations,processor(s) of the client device: train, based on a plurality oftraining instance, an on-device text-to-speech (TTS) generator modelthat is a portion of a generative adversarial network (GAN) model storedlocally at the client device (e.g., in RAM and/or ROM of the clientdevice). As described in detail herein, the on-device TTS generatormodel can be trained to include voice characteristic(s) of a user of theclient device. Further, subsequent to training the on-device TTSgenerator model, processor(s) of the client device: identify a textualsegment stored locally at the client device, process the textual segmentusing the trained on-device TTS generator model to generate synthesizedspeech audio data that includes synthesized speech corresponding to thetextual segment; process the synthesized speech audio data using theon-device ASR model to generate predicted ASR output; and generate agradient based on comparing the predicted ASR output to ground truthoutput corresponding to the textual segment. By utilizing the trainedon-device TTS generator model in generating the synthesized speech audiodata utilized in generating the gradient, the on-device ASR model can beupdated using gradient(s) that are generated based on synthesized speechaudio data that includes the voice characteristic(s) of the user of theclient device and that are based on textual segments that are likely tobe encountered at the client device, thereby improving performance ofthe on-device ASR model. For instance, by updating the ASR model inthese and other manners, the on-device ASR model can be personalized torecognize these textual segments that are likely to be encountered atthe client device, but otherwise are unlikely to be encountered atrespective client devices of other users, and therefore are moredifficult to recognize without such personalization of the on-device ASRmodel.

For example, the on-device ASR model can be an end-to-end speechrecognition model that is used to generate predicted ASR output of apredicted textual segment, and generating the gradient can be based oncomparing the predicted textual segment to a ground truth textualsegment corresponding to the ground truth output. Also, for example, theon-device ASR model can instead be used to generate predicted ASR outputof a sequence of predicted phonemes, and generating the gradient can bebased on comparing the sequence of predicted phonemes to a ground truthsequence of phonemes corresponding to the ground truth output.

In some implementations, the generated gradient is used, by one or moreprocessor(s) of the client device, to update one or more weights of theon-device ASR model based on the generated gradient. For example,backpropagation and/or other technique(s) can be used to update localweights of the on-device ASR model based on the gradient. This canimprove speech recognition performance, at the client device using theon-device ASR model, for spoken utterances that include the textualsegment. Moreover, this enables the on-device ASR model to be trainedbased on a particular textual segment, without requiring any actualhuman utterances of the particular textual segment (since the TTSgenerator model is used to generate synthesized speech of the particulartextual segment). Thus, the particular textual segment can be correctlyrecognized using the on-device ASR model, despite the textual segmentpotentially not having been included in any spoken utterance previouslydetected at the client device. Further, many implementations repeat thisprocess for a large quantity of textual segments stored locally at theclient device, thereby improving on-device speech recognitionperformance for spoken utterances that include any of the textualsegments. This effectively tailors the on-device ASR model to actualtextual segments that are stored (transiently or non-transiently) on thedevice, and that are likely to be included in spoken utterances directedto the client device.

In some implementations, the on-device ASR model that is updated basedon generated gradient(s) can be in addition to an on-device ASR modelthat is currently being utilized, by the client device, in performingspeech recognition of spoken utterances detected at the client device.In those implementations, the on-device ASR model that is updated can,in response to one or more conditions being satisfied, be deployed toeffectively replace the on-device ASR model that is currently beingutilized (thereby becoming the on-device ASR model that is currentlybeing utilized). For example, the condition(s) can include: on-devicevalidation of the on-device ASR model that is updated; on-devicedetermination that the on-device ASR model that is updated performsbetter (e.g., with respect to precision and/or recall) than theon-device ASR model currently being utilized; and/or occurrence of atleast a threshold quantity and/or duration of training of the on-deviceASR model that is updated. For example, determining that the on-deviceASR model that is updated performs better than the on-device ASR modelcurrently being utilized can be based on comparing performances based ontraining instance(s) that are generated according to techniquesdescribed herein, but that have not been utilized in training (i.e.,instead held back for testing). As another example, determining that theon-device ASR model that is updated performs better than the on-deviceASR model currently being utilized can be based on downloading testutterances at the client device (e.g., from the remote system), andprocessing the test utterances, using the updated on-device ASR model,to validate that the updated on-device ASR model has not diverged.Deploying a currently utilized on-device ASR model to effectivelyreplace an updated on-device ASR model can include updating weights ofthe currently utilized on-device ASR model with weights of the updatedon-device ASR model, or replacing the currently utilized on-device ASRmodel with the updated on-device ASR model. Once an updated on-deviceASR model effectively replaces a currently utilized on-device ASR modeland is used for speech recognition, a copy of the updated on-device ASRmodel can then be used as the new on-device ASR model to be updated.

In some implementations, the generated gradient is additionally oralternatively transmitted, by the client device and over a network, to aremote system. In those implementations, the remote system utilizes thegenerated gradient, and additional gradients from additional clientdevices and/or from the remote system, to update global weights of aglobal ASR model. The additional gradients from the additional clientdevices can be similarly generated, locally at the correspondingadditional client devices, based on corresponding locally stored textualsegments and locally generated synthesized speech thereof. In variousimplementations, the client device transmits the generated gradientwithout transmitting any of: the textual segment, the synthesized speechaudio data, the predicted ASR output, and the ground truth outputcorresponding to the textual segment. The remote system can utilize thegenerated gradient in updating the global ASR model, without anyreference to or use of the textual segment, the synthesized speech audiodata, the predicted ASR output, and the ground truth corresponding tothe textual segment. The transmitting of only the gradient can, in someinstances, utilize less network resources than transmitting of thelarger data size combination of the textual segment, the synthesizedspeech audio data, the predicted ASR output, and the ground truth outputcorresponding to the textual segment. Further, transmitting of thegradient preserves privacy and security of the on-device textualsegment, as the textual segments and the synthesized speech audio dataare not derivable from the gradient. In some implementations, one ormore differential privacy techniques (e.g., adding gaussian noise) canbe utilized to further ensure the textual segments and/or thesynthesized speech audio data are not derivable from the gradient.

In implementations where the remote system updates the global weights ofthe global ASR model, the remote system can thereafter provide theupdated global weights to client devices to cause the client devices toreplace the local weights of their on-device ASR models with the updatedglobal weights of the global ASR model. In some implementations, theremote system can additionally or alternatively provide the updatedglobal ASR model to client devices to cause the client devices toreplace their on-device ASR models with the updated global ASR model.On-device speech recognition performance is therefore improved throughutilization of the updated global weights or the updated global ASRmodel.

Some additional or alternative implementations disclosed herein aredirected to improving performance of speech synthesis that is performedlocally, at a client device, utilizing an on-device text-to-speech (TTS)generator model stored locally at the client device. In thoseimplementations, processor(s) of the client device: identify a textualsegment stored locally at the client device; process the textual segmentusing a trained on-device TTS generator model stored locally at theclient device to generate synthesized speech audio data that includessynthesized speech corresponding to the textual segment; and process thesynthesized speech audio data using a trained on-device TTSdiscriminator model stored locally at the client device to generatediscriminator output; and generate a gradient based on the discriminatoroutput (e.g., by comparing it to a ground truth output). Thediscriminator output indicates a prediction as to whether thesynthesized speech corresponds to (i) the synthesized speech audio datagenerated by the on-device TTS generator model or instead to (ii) anactual spoken utterance of a user of the client device. For example, theon-device TTS generator model and the on-device TTS discriminator modelcan be portions of a GAN model. The on-device TTS generator model triesto make the on-device TTS discriminator model predict that thesynthesized speech audio data is actually a spoken utterance of the userof the client device. As a result, the prediction made by the on-deviceTTS discriminator model can be compared to a ground truth label thatindicates the synthesized speech audio data was actually generated bythe on-device TTS generator model.

In some implementations, the generated gradient is used, by one or moreprocessor(s) of the client device, to update one or more weights of theon-device TTS generator model based on the generated gradient. Forexample, backpropagation and/or other technique(s) can be used to updatelocal weights of the on-device TTS generator model based on thegradient. This can improve speech synthesis performance, at the clientdevice using the on-device TTS generator model, for spoken utterancesthat include the textual segment. Moreover, this enables the on-deviceTTS generator model to be trained based on a particular textual segment,without requiring any actual human utterances of the particular textualsegment. Thus, the particular textual segment can be correctlysynthesized using the on-device TTS generator model, despite the textualsegment potentially not having been included in any synthesized speechpreviously generated at the client device. Further, many implementationsrepeat this process for a large quantity of textual segments storedlocally at the client device, thereby improving on-device speechsynthesis performance for spoken utterances that include any of thetextual segments. This effectively tailors the on-device TTS generatormodel to actual textual segments that are stored (transiently ornon-transiently) on the device, and that are likely to be included insynthesized speech generated at the client device.

In some implementations, the on-device TTS generator model that isupdated based on generated gradient(s) can be in addition to anon-device TTS generator model that is currently being utilized, by theclient device, in performing speech synthesis at the client device. Inthose implementations, the on-device TTS generator model that is updatedcan, in response to one or more conditions being satisfied, be deployedto effectively replace the on-device TTS generator model that iscurrently being utilized (thereby becoming the on-device TTS generatormodel that is currently being utilized). These conditions can be thesame or similar as those described above with respect to updating theon-device ASR model. Deploying a currently utilized on-device TTSgenerator model to effectively replace an updated on-device TTSgenerator model can include updating weights of the currently utilizedon-device TTS generator model with weights of the updated on-device TTSgenerator model, or replacing the currently utilized on-device TTSgenerator model with the updated on-device TTS generator model. Once anupdated on-device TTS generator model effectively replaces a currentlyutilized on-device TTS generator model and is used for speech synthesis,a copy of the updated on-device TTS generator model can then be used asthe new on-device TTS generator model to be updated.

In some implementations, the generated gradient is additionally oralternatively transmitted, by the client device and over a network, to aremote system. In those implementations, the remote system utilizes thegenerated gradient, and additional gradients from additional clientdevices, to update global weights of a global TTS generator model. Theadditional gradients from the additional client devices can be similarlygenerated, locally at the corresponding additional client devices, basedon corresponding locally stored textual segments and locally generatedsynthesized speech thereof. In various implementations, the clientdevice transmits the generated gradient without transmitting any of: thetextual segment, the synthesized speech audio data, and the predictionmade at the client device. The remote system can utilize the generatedgradient in updating the global TTS generator model, without anyreference to or use of the textual segment, the synthesized speech audiodata, and the prediction made at the client device. The transmitting ofonly the gradient utilizes less network resources than transmitting ofthe larger data size combination of the textual segment, the synthesizedspeech audio data, and the prediction made at the client device.Further, transmitting of the gradient preserves privacy and security ofthe on-device textual segment, as the textual segments and thesynthesized speech audio data are not derivable from the gradient. Insome implementations, one or more differential privacy techniques (e.g.,adding gaussian noise) can be utilized to further ensure the textualsegments and/or the synthesized speech audio data are not derivable fromthe gradient.

In implementations where the remote system updates the global weights ofthe global TTS generator model, the remote system can thereafter providethe updated global weights to client devices to cause the client devicesto replace the local weights of their on-device TTS generator modelswith the updated global weights of the global TTS generator model. Insome implementations, the remote system can additionally oralternatively provide the updated global TTS generator model to clientdevices to cause the client devices to replace their on-device TTSgenerator models with the updated global TTS generator model. On-devicespeech synthesis performance is therefore improved through utilizationof the updated global weights or the updated global TTS generator model.

Various techniques can be utilized by a client device to identifylocally stored textual segment(s) for utilization in generatinggradients based on the textual segment(s) and corresponding locallygenerated synthesized speech. For example, the textual segment(s) can beidentified based on them being included in a contacts list, a mediaplaylist, a list of aliases of linked smart devices (e.g., aliases ofsmart lights, smart plugs, and/or other smart devices linked with anaccount of the client device), from typed input received at the clientdevice, and/or from spoken utterances received at the client device. Asanother example, the textual segment(s) can additionally oralternatively be identified based on determining that the textualsegment(s) are out of vocabulary (i.e., textual segment(s) on which theon-device ASR model has not been previously trained). As yet anotherexample, a textual segment can be identified based on determining that aprior human utterance, detected via microphone(s) of the client device,included the textual segment and determining that a prior speechrecognition of the prior human utterance, performed using the on-deviceASR model, failed to correctly recognize the textual segment.Determining that the prior speech recognition failed to correctlyrecognize the textual segment can be based on received user input thatcancels an incorrect prediction that is based on the prior speechrecognition. Further, determining that the prior human utteranceincluded the textual segment can be based on the received user inputand/or based on additional received user input that is received afterthe user input that cancels the incorrect prediction based on the priorspeech recognition.

To conserve limited battery, processor, memory, and/or other resourcesof a client device, in various implementations a client device performsone or more steps disclosed herein only responsive to determining acurrent state of the client device satisfies one or more conditions. Forexample, generating the synthesized speech audio data, and/or processingthe synthesized speech audio data to generate the predicted ASR output,and/or generating the gradient, and/or updating the one or more weightscan be performed responsive to determining that the current state of theclient device satisfies the one or more conditions. Whether the one ormore conditions are satisfied can be determined based on sensor datafrom one or more sensors of the client device. The one or moreconditions can include, for example, that the client device is charging,that the client device has at least a threshold state of charge, that atemperature of the client device (based on one or more on-devicetemperature sensors) is less than a threshold, that the client device isnot being held by a user, and/or that the client device is connected toan unmetered network (e.g., WiFi).

In various implementations, the on-device TTS generator model and theon-device TTS discriminator model portions of the GAN model may betrained prior to generating the gradient(s). In some versions of thoseimplementations, the on-device TTS discriminator model can be trainedbased on a plurality of discriminator training instances. Each of thediscriminator training instances can include training instance input andtraining instance output. The training instance input can include audiodata that includes a spoken utterance of the user of the client deviceor synthesized speech audio data generated using the on-device TTSgenerator model (or another TTS generator model), and the traininginstance output can include a ground truth label that indicates whetherthe corresponding training instance input corresponds to the audio datathat includes the spoken utterance of the user of the client device(e.g., a semantic label of “real” or “human”, or a ground truthprobability corresponding thereto, such as “0” or “1”) or thesynthesized speech audio data generated using the on-device TTSgenerator model (or another TTS generator model) (e.g., a semantic labelof “fake” or “synthesized”, or a ground truth probability correspondingthereto, such as “0” or “1”). The training instances that include theaudio data that includes the spoken utterance of the user of the clientdevice may be considered positive discriminator training instances, andthe training instances that include the synthesized speech audio datagenerated using the on-device TTS generator model (or another TTSgenerator model) can be considered negative discriminator traininginstances. Further, the on-device TTS discriminator model, inprocessing, the training instance input, of a given discriminatortraining instance, predicts whether the training instance inputcorresponds to human speech or synthesized speech. The prediction caninclude, for example, a semantic label (e.g., “real”, “human”, “fake”,“synthesized”, etc.), a binary value (e.g., “0” for “synthesized” or“fake”, and “1” for “real” or “human”), and/or a probability (e.g.,“0.6” for “synthesized” or “fake”, and “0.4” for “real” or “human”).Moreover, the prediction can be compared to the ground to the groundtruth output to generate a loss, and the on-device TTS discriminatormodel can be updated based on the loss (e.g., backpropagated across theon-device TTS discriminator model to update weights thereof). This canbe repeated for a plurality of additional discriminator traininginstances to train the on-device TTS discriminator model.

In some versions of those implementations, the on-device TTS generatormodel can be trained based on a plurality of generator traininginstances. Each of the generator training instances can include traininginstance input and training instance output. The training instance inputcan include textual segment(s) (e.g., identified in any manner describedherein), and the training instance output can include a ground truthlabel that indicates any resulting audio data processed by the on-deviceTTS discriminator model corresponds to synthesized speech audio datagenerated using the on-device TTS generator model (e.g., a semanticlabel of “fake” or “synthesized”, or a ground truth probabilitycorresponding thereto, such as “0” or “1”). Further, the on-device TTSgenerator model, in processing the training instance input, of a givengenerator training instance, generates synthesized speech audio datathat includes synthesized speech, and the on-device TTS discriminatormodel, in processing the synthesized speech audio data, predicts whetherthe training instance input corresponds to human speech or synthesizedspeech. The prediction can include, for example, a semantic label (e.g.,“real”, “human”, “fake”, “synthesized”, etc.), a binary value (e.g., “0”for “synthesized” or “fake”, and “1” for “real” or “human”), and/or aprobability (e.g., “0.6”). Moreover, the prediction can be compared tothe ground truth output to generate a loss, and the on-device TTSgenerator model can be updated based on the loss (e.g., the loss may bebackpropagated across the on-device TTS generator model to updateweights thereof). This can be repeated for a plurality of additionalgenerator training instances to train the on-device TTS generator model.The loss utilized in updating the on-device TTS generator model may beconsidered an adversarial loss. Notably, in generating the synthesizedspeech audio data, the on-device TTS generator model tries to trick theon-device TTS discriminator model into predicting that the synthesizedspeech audio data corresponds to human speech of the user of the clientdevice. Thus, by updating the on-device TTS generator model based on theadversarial loss, the on-device TTS generator model can learn voicecharacteristics that reflect those of the user of the computing devicesince the on-device TTS discriminator model was trained to discriminatorbetween human speech of the user of the client device and synthesizedspeech generated by the on-device TTS generator model (or another TTSgenerator model).

In some additional or alternative versions of those implementations, thetraining instance output, for one or more of the generator traininginstances, can additionally or alternatively include ground truth audiodata corresponding to the textual segment(s) of the correspondingtraining instance input. In some further versions of thoseimplementations, acoustic features of the predicted synthesized speechaudio data generated using the on-device TTS generator model can becompared to acoustic features of the ground truth audio data, and anadditional loss can be generated based on comparing the acousticfeatures. The acoustic features can include, for example, audiowaveforms, mel-frequency cepstral coefficients (MFCCs), mel-filterbankfeatures, values associated with one or more prosodic properties, neuralrepresentations of the audio data (e.g., respective embeddings of theground truth audio data and the predicted synthesized speech audiodata), and/or other acoustic features of the synthesized speech audiodata and the ground truth audio data that can be compared. The on-deviceTTS generator model can additionally or alternatively be updated basedon the additional loss (e.g., the additional loss may be backpropagatedacross the on-device TTS generator model to update weights thereof).

As described herein, after updating of an on-device ASR model and/or anon-device TTS generator model according to implementations disclosedherein, the on-device ASR model can be utilized in processing audio datacorresponding to spoken utterances, from user(s) of the correspondingdevice, to generate corresponding predicted ASR output, and/or theon-device TTS model can be utilized in processing textual segments, fromclient device(s) of the user(s), to generate corresponding synthesizedspeech audio data that includes synthesized speech. In implementationswhere the on-device ASR model is utilized, a gradient can be generated,on-device, based on comparing an instance of a predicted ASR output toan instance of a ground truth output. The instance of the ground truthtextual segment can be determined, on-device, based on one or moreaction(s) and/or inaction(s) of the user responsive to content renderedat the device based on the instance of the predicted ASR output and/orbased on action(s) taken at the device based on the instance of thepredicted ASR output. For example, if the user confirms a predictedtextual segment, the predicted textual segment can be considered aground truth textual segment. For instance, if the spoken utterance is“call Francis”, and the predicted textual segment is “call Francis”, theuser can confirm the predicted textual segment by not cancelling aresulting dialing of a phone number for a contact named “Francis”. Also,for instance, if the spoken utterance is “call Francis”, the predictedtextual segment is “call Francis”, a prompt of “do you want to callFrancis” can be provided with a selectable “yes” option, and the usercan confirm the recognized text by selecting the “yes” option. Asanother example, if the user modifies the predicted textual segment(e.g., by adding and/or deleting character(s); and/or by deleting it andreplacing it with alternate text), the modified text can be consideredthe ground truth textual segment. For instance, if the spoken utteranceof “Hi Françoise, please call me soon” is provided for inclusion in atext message, and the incorrectly recognized text of “Hi Francis, pleasecall me soon” is incorporated into the text message, the user can select“Francis” and replace it with “Françoise”. Responsive to the userselecting “Francis” and replacing it with “Françoise”, the modified textof “Hi Françoise, please call me soon” can be utilized as the groundtruth textual segment. Also, for instance, if the spoken utterance of“Hi Françoise, please call me soon” is provided for inclusion in a textmessage, a first selectable graphical element of “Francis” and a secondselectable graphical element of “Françoise” can be presented to the userthat provided the spoken utterance. Responsive receiving a selection ofthe second graphical element of “Françoise” from the user, the text of“Hi Françoise, please call me soon” can be utilized as the ground truthtextual segment.

In implementations where the on-device TTS generator model is utilized,a gradient can be generated, on-device, based on processing, using theon-device TTS discriminator model, predicted synthesized speech audiodata to predict whether synthesized speech (included in the synthesizedspeech audio data) is actual spoken utterance(s) of the user(s) of theclient device(s) or the synthesized speech generated by on-device TTSgenerator model. The predicted synthesized speech audio data can begenerated based on textual segment(s) identified as described herein.The prediction made by processing the synthesized speech audio datausing the on-device TTS discriminator model can be compared to a groundtruth label that indicates the synthesized speech audio data. Forinstance, if the on-device TTS discriminator model incorrectly predictsthat the synthesized speech corresponds to actual spoken utterance(s) ofthe user(s) of the client device(s), then a first gradient can begenerated. However, if the on-device TTS discriminator model correctlypredicts that the synthesized speech corresponds to synthesized speechgenerated by the on-device TTS generator model, then a second gradientcan be generated.

Gradient(s) generated based on these techniques can be transmitted, bythe client device and over a network, to a remote system. In thoseimplementations, the remote system utilizes the generated gradient(s),and additional gradients generated from additional client devices in asimilar manner (e.g., after local updating of on-device model(s), suchas an on-device ASR model or an on-device TTS generator model), toupdate global weights of a global model(s) (e.g., a global ASR model ora global TTS generator model). It is noted that the updating of theglobal weights based on such gradients can occur along with, orindependent of, updating of the global weights based on gradients thatare based on locally stored textual segments and locally generatedsynthesized speech thereof. It is also noted that transmitting suchgradients can occur without transmitting any of: audio data, synthesizedspeech audio data, textual segment(s), predicted ASR output(s), and/orprediction(s). The remote system can utilize the generated gradient inupdating the global model(s), without any reference to or use of theaudio data, synthesized speech audio data, textual segment(s), predictedASR output(s), and/or prediction(s). The transmitting of only thegradient utilizes less network resources, and preserves privacy andsecurity of data stored and/or generated locally at the client device.In some implementations, one or more differential privacy techniques canbe utilized to further ensure preservation of the privacy and securityof data stored and/or generated locally at the client device.

In some implementations, after updating of an on-device ASR model and/oran on-device TTS generator model according to implementations disclosedherein, biasing of the on-device speech recognition and/or on-devicespeech synthesis based on textual segment(s) can also be utilized, whenthe on-device ASR model is processing audio data corresponding to spokenutterances to generate corresponding predicted textual segments and/orwhen the on-device TTS generator model is processing textual segment(s)to generate corresponding predicted synthesized speech audio data. Forexample, the on-device speech recognition can be biased toward one ormore textual segments stored on the device, such as contact alias(es),road name(s), media name(s), and/or other textual segment(s). Thetextual segment(s) to which on-device speech recognition is biased canbe the same across multiple iterations of speech recognition and/or canvary amongst iterations. For example, biasing toward a first textualsegments can occur in first device context(s) (e.g., context(s) based onwhich application(s) are executing or in the foreground, based on timeof day, based on day of week, etc.) and biasing toward disparate secondtextual segments can occur in second device context(s). Biasing theon-device speech recognition to textual segment(s) can include, forexample, injecting the textual segment(s) into the speech recognition byboosting the probability of the textual segment(s) during decoding.Biasing the on-device speech recognition based on textual segment(s)after updating of the on-device speech recognition model according toimplementations disclosed herein can be more effective than biasingwithout such updating. This can be due to, for example, the on-devicespeech recognition model having been trained on sound sequences from thetextual segment(s) as a result of having been updated based onsynthesized speech that is based on the textual segment(s). As anotherexample, the on-device speech synthesis can be biased towards voicecharacteristic(s) of the user of the client device. The voicecharacteristic(s) of the user of the client device can include, forexample, prosodic properties include prosodic properties (e.g., one ormore of intonation, tone, stress, rhythm, tempo, or pause) that areindicative of speech of the user of the client device. Notably, althoughthe synthesized speech includes voice characteristic(s) of the user, thesynthesized speech need not be an exact match of speech of the user ofthe client device.

In various implementations, global model(s) described herein may beparticular to a specific geographic language and/or a specific language.For example, the gradient(s) generated at a client device that isprimarily located in a particular geographic region may only be utilizedto update global model(s) associated with the particular geographicregion. For instance, the gradient(s) generated at a client device thatis primarily located in Chicago, Ill. may only be utilized to the globalmodel(s) associated with a Midwest geographic region, or a United Statesgeographic region. As another example, the gradient(s) generated at aclient device that primarily utilizes the Spanish language may only beutilized to update global model(s) associated with the Spanish language.In this manner, the global model(s) may be one of N available model(s)for a given language and/or a particular geographic region.

Some implementations of client devices disclosed herein include anautomated assistant application that generates the on-device speechrecognitions (or that utilizes generated on-device speech recognitions)and/or that generates the on-device synthesized speech (or that utilizesgenerated on-device synthesized speech). The automated assistantapplication can be installed “on-top of” an operating system of theclient device and/or can itself form part of (or the entirety of) theoperating system of the client device. The automated assistantapplication includes, and/or has access to, the on-device speechrecognition and the on-device speech synthesis as well as optionallyon-device natural language understanding and/or on-device fulfillment.On-device natural language understanding (NLU) can be performed using anon-device NLU module that processes recognized text, generated using theon-device speech recognition, and optionally contextual data, togenerate NLU data. NLU data can include intent(s) that correspond to thespoken utterance and optionally parameter(s) (e.g., slot values) for theintent(s). On-device fulfillment can be performed using an on-devicefulfillment module that utilizes the NLU data (from the on-device NLU),and optionally other local data, to determine action(s) to take toresolve the intent(s) of the spoken utterance (and optionally theparameter(s) for the intent). This can include determining local and/orremote responses (e.g., answers) to the spoken utterance, interaction(s)with locally installed application(s) to perform based on the spokenutterance, command(s) to transmit to Internet-of-things (IoT) device(s)(directly or via corresponding remote system(s)) based on the spokenutterance, and/or other resolution action(s) to perform based on thespoken utterance. The on-device fulfillment can then initiate localand/or remote performance/execution of the determined action(s) toresolve the spoken utterance.

In various implementations, remote speech processing, remote NLU, remotefulfillment, and/or remote speech synthesis can at least selectively beutilized. For example, recognized text can at least selectively betransmitted to remote automated assistant component(s) for remote NLUremote fulfillment, and/or remote speech synthesis. For instance, therecognized text can optionally be transmitted for remote performance inparallel with on-device performance, or responsive to failure ofon-device NLU and/or on-device fulfillment. However, on-device speechprocessing, on-device NLU, on-device fulfillment, on-device speechsynthesis, and/or on-device execution can be prioritized at least due tothe latency reductions they provide when resolving a spoken utterance(due to no client-server roundtrip(s) being needed to resolve the spokenutterance). Further, on-device functionality can be the onlyfunctionality that is available in situations with no or limited networkconnectivity.

The above description is provided as an overview of some implementationsof the present disclosure. Further description of those implementations,and other implementations, are described in more detail below.

Some implementations disclosed herein include one or more computingdevices that include one or more processors such as central processingunit(s) (CPU(s)), graphics processing unit(s) (GPU(s)), digital signalprocessor(s) (DSP(s)), and/or tensor processing unit(s) (TPU(s)). One ormore of the processors are operable to execute instructions stored inassociated memory, and the instructions are configured to causeperformance of any of the methods described herein. The computingdevices can include, for example, client assistant devices withmicrophone(s), at least one display, and/or other sensor component(s).Some implementations also include one or more non-transitory computerreadable storage media storing computer instructions executable by oneor more processors to perform any of the methods described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A depicts an example process flow for updating an on-device ASRmodel based on a gradient generated locally at a client device and/ortransmitting the gradient to a remote system for updating a global ASRmodel, in accordance with various implementations.

FIG. 1B depicts an example process flow for updating an on-device TTSgenerator based on a gradient generated locally at a client deviceand/or transmitting the gradient to a remote system for updating aglobal TTS generator model, in accordance with various implementations.

FIG. 2 is a block diagram of an example environment that includesvarious components from FIGS. 1A and 1B, and in which implementationsdisclosed herein may be implemented.

FIG. 3 depicts a flowchart illustrating an example method of training anon-device TTS generator model stored locally at a client device, inaccordance with various implementations.

FIG. 4 depicts a flowchart illustrating an example method of generatinga gradient, locally at a client device, and utilizing the generatedgradient to update weight(s) of an on-device ASR model and/ortransmitting the gradient to a remote system, in accordance with variousimplementations.

FIG. 5 depicts a flowchart illustrating an example method of generatinga gradient, locally at a client device, and utilizing the generatedgradient to update weight(s) of an on-device TTS generator model and/ortransmitting the gradient to a remote system, in accordance with variousimplementations.

FIG. 6 depicts a flowchart illustrating an example method oftransmitting, to a remote system, gradient(s) generated locally at aclient device that is performing corresponding instances of the methodof FIG. 4 and/or the method of FIG. 5 , in accordance with variousimplementations.

FIG. 7 depicts a flowchart illustrating an example method of updatingweight(s) of global model(s) based on gradient(s) received from remoteclient device(s) that are performing corresponding instances of themethod of FIG. 4 and/or the method of FIG. 5 , and transmitting, toremote client device(s), the updated weight(s) and/or the updated globalmodel(s), in accordance with various implementations.

FIG. 8 illustrates an example architecture of a computing device, inaccordance with various implementations.

DETAILED DESCRIPTION

Turning now to FIG. 1A, an example process flow for updating anon-device ASR model based on a gradient generated locally at a clientdevice and/or transmitting the gradient to a remote system for updatinga global ASR model is depicted. A client device 110 is illustrated inFIG. 1A, and includes the components that are encompassed within the boxof FIG. 1A that represents client device 110. A segment identifier 120of the client device 110 accesses on-device storage 111 to identify atextual segment 101A. The on-device storage 110 can include, forexample, read-only memory (ROM) and/or random-access memory (RAM). Thetextual segment 101A identified by the segment identifier 120 is atextual segment that is transiently or non-transiently stored inon-device storage 111. For example, the textual segment 101 can be: analias of a contact that is stored in a contacts list; a name of a roadthat is stored as an address in a contacts list; a name of a song orother media item that is included in a playlist of a media application;an alias of a smart device, where the alias is stored at the clientdevice and the smart device is associated with an account of the user; atextual segment typed via a virtual keyboard at the client device; atextual segment copied to a pasteboard at the client device; a textualsegment rendered by an application of the computing device (andoptionally identified using a screenshot and image recognition); atextual segment generated based on processing spoken utterances of auser of the client device 110; or other textual segment. The on-devicestorage 111 can include any textual segments corresponding to speech ofthe user of the client device 110 (e.g., determined using speakeridentification or speaker diarization techniques), or any other user ofthe client device 110 in instances where the client device 110 isassociated with multiple users (e.g., a shared client device located ina primary dwelling of a family).

In some implementations, the segment identifier 120 identifies thetextual segment 101 based on it being newly encountered or newly storedat the client device 110. For example, the segment identifier 120 canidentify the textual segment 101A based on it being included in a newlyadded contact, being an alias for a newly added smart device, being analias for a newly added song to a playlist, or being generated based onprocessing a spoken utterance of the user of the client device 110. Insome implementations, the segment identifier 120 identifies the textualsegment 101A based on determining that the textual segment 101 is out ofvocabulary, which can be based on determining that the textual segment101A is not included in a locally stored lexicon. Notably, any textualsegments identified by the segment identifier 120 may, in manyimplementations, be restricted to those derived from spoken utterancesof the user (or user(s)) of the client device 110.

In some implementations, the segment identifier 120 can identify thetextual segment 101A based on determining that a spoken utterance,detected via microphone(s) of the client device 110, included thetextual segment 101A and determining that a prior speech recognition ofthe prior spoken utterance failed to correctly recognize the textualsegment 101A. In those implementations, the segment identifier 120 candetermine that the prior speech recognition failed to correctlyrecognize the textual segment based on received user input, at theclient device 110, that cancels an incorrect prediction that is based onthe prior speech recognition. Further, the segment identifier 120 candetermine that the prior human utterance included the textual segment101A based on additional received user input, that is received after theuser input that cancels the incorrect prediction based on the priorspeech recognition.

The segment identifier 120 provides at least the textual segment 101A toan on-device TTS generator 122A. In some implementations, the segmentidentifier 120 provides the identified textual segment, as well asadditional textual segment(s), to the on-device TTS generator 122A. Forexample, the segment identifier 120 can append the additional textualsegment(s) before and/or after the textual segment, and provide thetextual segment 101A with appended additional textual segment(s) to theon-device TTS generator 122A. In some of those implementations, textualsegment generator 120A can process the textual segment 101A to generatea plurality of contextually relevant and semantically diverse additionaltextual segments. For example, the textual segment generator 120A candetermine that the additional textual segment(s) are semanticallydiverse based on generating, over an encoder model, a plurality ofcandidate textual segment embeddings. The generated candidate textualsegment embeddings can be a lower-dimensional representation mapping thecandidate textual segment(s) to a lower-dimensional candidate textualsegment embedding space. The embedding for a given one of the candidatetextual segment(s) may be compared to embedding(s) of candidate textualsegment(s), and the candidate textual segment(s) can be pre-pended orappended to the textual segment 101A if the comparing indicatesdifference metric(s) is satisfied. For example, the difference metric(s)may be satisfaction of a threshold that indicates a sufficient degree ofsemantic difference between the given textual segment and an alreadyselected candidate textual segment. As another example, the textualsegment generator 120A can determine that textual segment(s) arecontextually relevant based on defined relationship(s) of an additionaltextual segment to a particular corpus from which the textual segment101A was identified. For example, if textual segment 101A is identifiedfrom a media corpus, additional textual segments of “play” and “show me”can have a defined relationship to the media corpus, and one of thoseadditional textual segments appended before the textual segment. Asanother example, if the textual segment 101A is identified from a priorspoken utterance of “Send Francoisé a message”, but speech recognitionfailed to correctly recognize “Francoisé”, additional textual segmentsof “Schedule a meeting with Francoisé” and “Remind me to tell Francoiséabout the presentation” can be generated. Generating a plurality ofsemantically diverse and contextually relevant additional textualsegment(s) that are pre-pended or appended to the textual segment 101Aallows multiple diverse training instances to be generated based on thetextual segment 101A.

The on-device TTS generator 122A processes the textual segment 101A (andany pre-pended or appended additional textual segment(s)), using anon-device TTS generator model 152A1, to generate synthesized speechaudio data 102 that includes synthesized speech of the identifiedtextual segment 101A. For example, the on-device TTS generator 122A candetermine a sequence of phonemes determined to correspond to the textualsegment 101A (and any pre-pended or appended additional textualsegment(s)) and process the sequence of phonemes using the on-device TTSgenerator model 152A1, to generate synthesized speech audio data 102.The synthesized speech audio data 102 can be, for example, in the formof an audio waveform. In determining a sequence of phonemes thatcorrespond to the textual segment 101, the on-device TTS generator 122Acan access a tokens-to-phonemes mapping stored locally at the clientdevice 110, such as optional token-to-phonemes mapping 157. In someimplementations, the synthesized speech audio data 102 that is generatedcan be a mix of human speech and synthesized speech. For example, if thetextual segment 101A is identified from a prior spoken utterance of“Send Francoisé a message”, but speech recognition failed to correctlyrecognize “Francoisé”, additional textual segments of “Schedule ameeting with Francoisé” and “Remind me to tell Francoisé about thepresentation” can be generated. In these examples, the portion of thesynthesized speech audio data 102 for the additional textual segmentsmay include audio data that captures speech of the user for “Francoisé”,but the remaining portions of the synthesized speech audio data 102 maycorrespond to synthesized speech. For instance, assume that thesynthesized speech audio data 102 corresponds to “Schedule a meetingwith Francoisé”. In this example, the portion that corresponds to“Schedule a meeting with” can be synthesized speech audio data generatedusing the on-device TTS generator 122A, and the portion that correspondsto “Francoisé” can be audio data from the original spoken utterance.

In some implementations, the on-device TTS generator model 152A istransmitted (e.g., by the remote system 160 or other component) forstorage and use at the client device 110, based on a geographic regionof the user of the client device 110, a primary language of the user ofthe client device 110, and/or other properties of the client device 100and/or a user of the client device 110. For example, the on-device TTSgenerator model 152A1 can be one of N available TTS generator models fora given language, but can be trained based on spoken utterances that arespecific to a particular geographic region and provided to client device110 based on the client device 110 being primarily located in theparticular geographic region.

The on-device TTS generator model 152A1 is trained by the on-device TTSgenerator training engine 130A to adapt the on-device speech synthesismodel 152 to voice characteristic(s) of user(s) of the client device 110based on a plurality of training instances. Each training instance, ofthe plurality of training instances, includes training instance inputand training instance output.

In some implementations, the on-device TTS generator training engine130A can identify a ground truth transcription for a prior spokenutterance of the user of the client device 110 by generating atranscription (e.g., by processing the corresponding audio data usingthe on-device ASR model 154A), and identifies the transcription as theground truth transcription. The ground truth transcription can alsoinclude one or more ground truth textual segments for correspondingterms included in the prior spoken utterance of the user of the clientdevice 110. Identifying the transcription as “ground truth” canoptionally be contingent on a confidence measure for the transcriptionsatisfying a threshold and/or based on user action(s) (or inactions),responsive to generating the transcription, indicating the transcriptionis correct. In those implementations, the transcription (or a textualsegment included in the transcription) can be utilized as traininginstance input for a given training instance, and the correspondingaudio data that captures the prior spoken utterance can be utilized astraining instance output for the given training instance. In someversions of those implementations, the prior spoken utterance can be anenrollment phrase spoken by the user for text-independent ortext-dependent identification, and the enrollment phrase can be renderedto the user to inform the user what to speak during the enrollment. Inthose implementations, the enrollment phrase rendered to the user can beutilized as the ground truth transcription (or a ground truth textualsegment included in the transcription).

For example, prior to generating the synthesized speech audio data 102,the on-device TTS generator training engine 130A can identify audio datathat is detected via one or more microphones of the client device 110and that captures a prior human utterance. Further, the on-device TTSgenerator training engine 130A can identify a ground truth transcriptionfor the prior human utterance. Yet further, the on-device TTS generatortraining engine 130A can cause the on-device TTS generator 122A toprocess the ground truth transcription using the on-device TTS generatormodel 152A1 to generate prior synthesized speech audio data, and cangenerate a loss based on comparing the prior synthesized speech audiodata to the prior audio data. The on-device speech TTS generator engine130A can then update weight(s) of the on-device TTS generator model152A1 based on the loss (e.g., using backpropagation and/or othertraining technique(s)).

In some additional or alternative implementations, the on-device TTSgenerator model 152A1 may be a generator portion of a generativeadversarial network (GAN) model. The GAN model may also include anon-device TTS discriminator model 152A2 stored locally at the clientdevice 110. In some versions of those implementations, on-device TTSdiscriminator training engine 130B can identify a plurality of traininginstances utilized in training the on-device TTS discriminator model152A2 (e.g., discriminator training instances). The on-device TTSdiscriminator training engine 130B can identify a plurality of positivetraining instances and a plurality of negative training instances. Thetraining instance input, for each of the plurality of positive traininginstances, can include audio data that captures a prior spoken utteranceof the user of the client device 110. The training instance output, foreach of the plurality of positive training instances, can include aground truth label (e.g., a binary value, a semantic label, aprobability, etc.) that indicates the training instance inputcorresponds to speech of the user of the client device 110. In contrast,the training instance input, for each of the plurality of negativetraining instances, can include synthesized audio data that capturessynthesized speech generated using the on-device TTS generator model152A1 (or another speech synthesizer stored locally at the client deviceor remotely at a remote system). The training instance output, for eachof the plurality of negative training instances, can include a groundtruth label (e.g., another binary value, another semantic label, aprobability, etc.) that indicates the training instance inputcorresponds to synthesized speech.

For example, prior to generating the synthesized speech audio data 102,the on-device TTS discriminator training engine 130B can identify audiodata that is detected via microphone(s) of the client device 110 andthat captures a prior spoken utterance of the user of the client device110. Further, the on-device TTS discriminator training engine 130B canidentify synthesized speech audio data that captures synthesized speechgenerated using the on-device TTS generator model 152A1 (or anotherspeech synthesizer stored locally at the client device or remotely at aremote system). Yet further, the on-device TTS discriminator trainingengine 130B can identify corresponding ground truth labels for thespoken utterance and the synthesized speech. Moreover, the on-device TTSdiscriminator training engine 130B can cause the on-device TTSdiscriminator 1228 to process the audio data (or synthesized speechaudio data) using the on-device TTS discriminator model 152A2 to predictwhether the audio data (or synthesized speech audio data) corresponds toa spoken utterance (or portion thereof) of the user of the client device110 or synthesized speech generated using the on-device TTS generatormodel 152A1 (or another speech synthesizer stored locally at the clientdevice or remotely at a remote system), and can generate a loss based onthe prediction. The on-device TTS discriminator training engine 130B canthen update weight(s) of the on-device TTS discriminator model 152A2based on the loss (e.g., using backpropagation and/or other trainingtechnique(s)).

Further, the on-device TTS generator training engine 130A can identify aplurality of training instances utilized in training in the on-deviceTTS generator model 152A1 (e.g., generator training instances). Thetraining instances can include a given textual segment stored locally atthe client device 110 (and any pre-pended or appended additional textualsegment(s)). For example, prior to generating the synthesized speechaudio data 102 (and optionally subsequent to training the on-device TTSdiscriminator model 152A2), the on-device TTS generator training engine130A can identify a given textual segment stored locally on the clientdevice 110. The on-device TTS generator training engine 130A can causethe on-device TTS generator engine 122A to process the given textualsegment (and any pre-pended or appended additional textual segment(s))using the on-device TTS generator model 152A1 to generate synthesizedspeech audio data that includes synthesized speech. Further, theon-device TTS discriminator training engine 130B can cause the on-deviceTTS discriminator 1228 to process the synthesized speech audio datausing the on-device TTS discriminator model 152A2 to predict whether thesynthesized speech audio data corresponds to a spoken utterance (orportion thereof) of the user of the client device 110 or synthesizedspeech generated using the on-device TTS generator model 152A1, and cangenerate a loss based on the prediction. The on-device TTS generatortraining engine 130A can then update weight(s) of the on-device TTSgenerator model 152A1 based on the loss (e.g., using backpropagationand/or other training technique(s)). This loss may be considered anadversarial loss utilized in training the on-device TTS generator model152A1 of the GAN model. In other words, in training the on-device TTSgenerator model 152A1, the on-device TTS generator model 152A1 tries totrick the on-device TTS generator model 15281 into predicting thesynthesized speech audio data corresponds to audio data that captures aspoken utterance of the user of the client device 110 rather thansynthesized speech generated using the on-device TTS generator model152A1.

The trained on-device TTS generator 122A provides the synthesized speechaudio data 102 to the on-device ASR engine 124. The on-device ASR engine124 processes the synthesized speech audio data 102, using an on-deviceASR model 154A, to generate predicted ASR output (e.g., a predictedtextual segment 103A, a predicted sequence of phonemes 103B, and/orother predicted ASR output(s)).

For example, when the on-device speech ASR model 154A is an end-to-endspeech recognition model, the on-device ASR engine 124 can generate thepredicted textual segment 103A directly using the model. For instance,the on-device speech ASR model 154A can be an end-to-end model used togenerate predicted text on a character-by-character basis (or anothertoken-by-token basis). One non-limiting example of such an end-to-endmodel used to generate predicted text on a character-by-character basisis a recurrent neural network transducer (RNN-T) model. An RNN-T modelis a form of sequence-to-sequence model that does not employ attentionmechanisms. Unlike most sequence-to-sequence models, which typicallyneed to process the entire input sequence (e.g., an audio data waveform,or mel-frequency cepstral coefficients (MFCCs) or other representation)to produce an output the predicted textual segment(s), an RNN-T modelcan be used to continuously process input samples and stream outputsymbols (e.g., characters of the alphabet).

Also, for example, when the on-device ASR model 154A is not anend-to-end speech recognition model, the on-device ASR engine 124 caninstead generate predicted phonemes 103B (and/or other representations).For instance, with such models the predicted phonemes 103B (and/or otherrepresentations) are then utilized by the on-device ASR engine 124 todetermine predicted textual segment(s) that conform to the sequence ofphonemes. In doing so, the on-device ASR engine 124 can optionallyemploy a decoding graph, a lexicon, and/or other resource(s).

When the on-device ASR engine 124 generates the predicted textualsegment 103A, it is provided to gradient engine 126. Gradient engine 126compares the predicted textual segment 103A to the textual segment 101Ato generate a gradient 105. The gradient may be derived from a lossfunction used to train the model, such that the gradient represents avalue of that loss function (or a derivative thereof) obtained fromcomparison of the ground truth output to the predicted output. Forexample, when the predicted textual segment 103A and the textual segment101A match, the gradient engine 126 can generate a zero gradient. Also,for example, when the predicted textual segment 103A and the textualsegment 101 do not match, the gradient engine 126 can generate anon-zero gradient that is optionally dependent on the extent of themismatching. The extent of the mismatching can be based on an extent ofmismatching between characters of the textual segments, an extent ofphoneme mismatching between the textual segments, and/or based on otherdeterministic comparisons. As one non-limiting particular example, ateach iteration of generating the predicted textual segment 103A, theon-device ASR engine 124 can generate a corresponding probability foreach of a plurality of characters, and select the highest probabilitycharacter as the “next” character. The gradient engine 126 can, in suchan example, determine a gradient based on comparing the predictedprobabilities at each iteration to ground truth probabilities for eachiteration (i.e., where the ground truth character at each iteration isthe “next” character in the textual segment 101A and is assigned a “1”probability, and all others a “0” probability).

When the on-device ASR engine 124 generates the predicted phonemes 103B,they are provided to gradient engine 126. Gradient engine 126 comparesthe predicted phonemes 103N to ground truth sequence of phonemes 104determined to correspond to the textual segment 101A. In someimplementations, phoneme engine 127 can access tokens-to-phonemesmapping 157 to determine the ground truth sequence of phonemes 104 thatcorrespond to the textual segment 101A. As an example, when thepredicted phonemes 103B and the phonemes 104 match, the gradient engine126 can generate a zero gradient. As another example, when the predictedphonemes 103B and the phonemes 104 do not match, the gradient engine 126can generate a non-zero gradient that is optionally dependent on theextent of the mismatching. The extent of the mismatching can be based ona quantity of mismatched phonemes, a quantity of incorrectly orderedphonemes, and/or a distance (in phoneme space) between mismatchedphoneme(s), and/or based on other deterministic comparisons.

It is noted that in implementations where additional textual segment(s)is pre-pended and/or appended to the textual segment 101A as describedherein, the predicted textual segment 103A will also include aprediction of the pre-pended and/or appended additional textualsegment(s). The prediction of the pre-pended and/or appended additionalsegment(s) can be ignored in generating the gradient (e.g., term(s)discarded that correspond to the appended additional segment) or,alternatively, the pre-pended and/or appended additional segment(s) canbe considered in generating the gradient (e.g., the prediction can becompared to the textual segment with the appended additional textualsegment).

The gradient engine 126 provides the gradient 105 to on-device ASRtraining engine 128 and/or transmits the gradient 105 to remote system160. The on-device ASR training engine 128, when it receives thegradient 105, uses the gradient 105 to update the on-device ASR model154A. For example, the on-device ASR training engine 128 can utilizebackpropagation and/or other techniques to update the on-device ASRmodel 154A. It is noted that, in some implementations, the on-device ASRtraining engine 128 can utilize batch techniques to update the on-deviceASR model 154A based on the gradient 105 and additional gradientsdetermined locally at the client device 110 on the basis of additionaltextual segments.

When the remote system 160 receives the gradient 105, a remote trainingengine 162 of the remote system uses the gradient 105, and additionalgradients 106 from additional client devices 170, to update globalweights of a global ASR model 154B. The additional gradients 106 fromthe additional client devices 170 can each be generated based on thesame or similar techniques as described above with respect to gradient105 in FIG. 1A (but on the basis of locally identified textual segments101A that are particular to those additional client devices 170).Although not depicted in FIG. 1A, the additional gradients canadditionally or alternatively be generated at the remote system 160based on the same or similar techniques as described above with respectto gradient 105 in FIG. 1A (but on the basis of remotely identifiedtextual segments 101A that are accessible by the remote system 160).

An update distribution engine 164 can, responsive to one or moreconditions being satisfied, provide, to client device 110 and/or otherclient device(s), the updated global weights and/or the updated globalASR model itself, as indicated by 107. The one or more conditions caninclude, for example, a threshold duration and/or quantity of trainingsince updated weights and/or an updated speech recognition model waslast provided. The one or more conditions can additionally oralternatively include, for example, a measured improvement to theupdated speech recognition model and/or passage of a threshold durationof time since updated weights and/or an updated speech recognition modelwas last provided. When the updated weights are provided to the clientdevice 110, the client device 110 can replace local weights, of theon-device ASR model 154A, with the updated weights. When the updatedglobal ASR model is provided to the client device 110, the client device110 can replace the on-device ASR model 154A with the updated global ASRmodel 154B.

By transmitting the gradient 105 to the remote system 160, and updatingthe global weights of the global ASR model 154B based on the gradient105, the on-device ASR model 154A can be utilized to update the globalASR model 154B. For example, assume the on-device ASR model 154A knowsthat the sound “fran-swaz” corresponds to a textual segment of“Françoise” since it was previously corrected at the client device 110and the on-device ASR model 154A was updated based on a gradientgenerated based on this correction. However, the global ASR model 154Bmay not know that the sound “fran-swaz” corresponds to a textual segmentof “Françoise”. Nonetheless, by updating the global ASR model 154B basedon a gradient that is generated based on this correction, the global ASRmodel 154B can also learn that the sound “fran-swaz” corresponds to atextual segment of “Françoise”, rather than “François” or “Francis”. Incontrast, if the audio data corresponding to the sound “fran-swaz” wasstored at the remote system 160 and annotated by a human, the human maynot know that “fran-swaz” corresponds to “Françoise” and incorrectlyannotate the sound “fran-swaz” as corresponding to “François” or“Francis” (e.g., a hard negative). In this example, the global ASR model154B may be updated to learn this error (e.g., to select the textualsegment “François” or “Francis” instead of Françoise” in response toreceiving audio data that captures the sound “fran-swaz”).

Turning now to FIG. 1B, an example process flow for updating anon-device TTS generator model based on a gradient generated locally at aclient device and/or transmitting the gradient to a remote system forupdating a global TTS generator model is depicted. The client device 110of FIG. 1A can additionally or alternatively perform instances of theprocess flow illustrated in FIG. 1B. In some implementations, and incontrast with FIG. 1A, the trained on-device TTS generator 122A providesthe synthesized speech audio data 102 directly to the gradient engine126 rather than to the on-device ASR engine 124. Gradient engine 126compares the synthesized speech audio data 102 to ground truth audiodata 101B. More particularly, the gradient engine 126 can compareacoustic features of the synthesized speech audio data 102 and theground truth audio data 101B. The acoustic features can include, forexample, audio waveforms, mel-frequency cepstral coefficients (MFCCs),mel-filterbank features, values associated with one or more prosodicproperties, neural representations of the audio data (e.g., respectiveembeddings of the ground truth audio data and the predicted synthesizedspeech audio data), and/or other acoustic features of the synthesizedspeech audio data and the ground truth audio data that can be compared.The ground truth audio data 101B can be stored in association with thetextual segment 101B processed by the on-device TTS generator 122A(e.g., in on-device storage 111), and can be identified along with thetextual segment 101A. The gradient may be derived from a loss functionused to train the model, such that the gradient represents a value ofthat loss function (or a derivative thereof) obtained from comparison ofthe ground truth output to the predicted output. For example, when theacoustic features of the synthesized speech audio data 102 and theground truth audio data 101B match, the gradient engine 126 can generatea zero gradient. Also, for example, when the acoustic features of thesynthesized speech audio data 102 and the ground truth audio data 101Bdo not match, the gradient engine 126 can generate a non-zero gradientthat is optionally dependent on the extent of the mismatching. Theextent of the mismatching can be based on an extent of mismatchingbetween characters of the textual segments, an extent of phonememismatching between the textual segments, and/or based on otherdeterministic comparisons.

In some additional or alternative implementations, and in contrast withFIG. 1A, the trained on-device TTS generator 122A provides thesynthesized speech audio data 102 to the on-device TTS discriminator122B rather than to the on-device ASR engine 124 (or directly to thegradient engine 126). The on-device TTS discriminator 122B can processthe synthesized speech audio data 102 using the on-device TTSdiscriminator model 152A2 to make a prediction 108. The prediction 108made by the on-device TTS discriminator model 152A2 can indicate whetherthe synthesized speech audio data 102 corresponds to a spoken utterance(or portion thereof) of the user of the client device 110 or synthesizedspeech generated using the on-device TTS generator model 152A1. Theon-device TTS discriminator 122B provides the prediction 108 to thegradient engine 126. The prediction 108 can be, for example, a binaryvalue (e.g., where “0” corresponds to synthesized speech and where “1”corresponds to a spoken utterance of the user of the client device 110),a semantic label (e.g., where “synthesized” or “fake” corresponds tosynthesized speech and where “human” or “real” corresponds to a spokenutterance of the user of the client device 110), and/or a probability(e.g., “0.65” associated with “synthesized” and “0.35” associated with“real”). Gradient engine 126 can generate the gradient 126 based on theprediction 108. For example, when the prediction 108 includes a binaryvalue or semantic label and the prediction 108 is correct, the gradientengine 126 can generate a zero gradient. Also, for example, whenprediction 108 includes a probability and the prediction 108 is notcorrect, the gradient engine 126 can generate a non-zero gradient thatis optionally dependent on the extent of the mismatching between theprobability and a ground truth probability associated with the correctprediction.

It is noted that in implementations where additional textual segment(s)is pre-pended and/or appended to the textual segment 101A as describedherein, the synthesized speech audio data 102 will also includesynthesized speech corresponding to the pre-pended and/or appendedadditional textual segment(s). The synthesized speech corresponding tothe pre-pended and/or appended additional segment(s) can be ignored ingenerating the gradient (e.g., term(s) discarded that correspond to theappended additional segment) or, alternatively, the pre-pended and/orappended additional segment(s) can be considered in generating thegradient (e.g., the prediction can be compared to the textual segmentwith the appended additional textual segment).

The gradient engine 126 provides the gradient 105 to on-device TTSgenerator training engine 130A and/or transmits the gradient 105 to theremote system 160. The on-device TTS generator training engine 130A,when it receives the gradient 105, uses the gradient 105 to update theon-device TTS generator model 152A1. For example, the on-device TTSgenerator training engine 130A can utilize backpropagation and/or othertechniques to update the on-device TTS generator model 152A1. It isnoted that, in some implementations, the on-device TTS generatortraining engine 130A can utilize batch techniques to update theon-device TTS generator model 152A1 based on the gradient 105 andadditional gradients determined locally at the client device 110 on thebasis of additional textual segments.

When the remote system 160 receives the gradient 105, a remote trainingengine 162 of the remote system uses the gradient 105, and additionalgradients 106 from additional client devices 170, to update globalweights of a global TTS generator model 152B. The additional gradients106 from the additional client devices 170 can each be generated basedon the same or similar technique as described above with respect togradient 105 in FIG. 18 (but on the basis of locally identified textualsegments 101A that are particular to those additional client devices170). Although not depicted in FIG. 1B, the additional gradients canadditionally or alternatively be generated at the remote system 160based on the same or similar techniques as described above with respectto gradient 105 in FIG. 1B (but on the basis of remotely identifiedtextual segments 101A that are accessible by the remote system 160).

An update distribution engine 164 can, responsive to one or moreconditions being satisfied, provide, to client device 110 and/or otherclient device(s), the updated global weights and/or the updated globalTTS generator model itself, as indicated by 107. The one or moreconditions can include, for example, a threshold duration and/orquantity of training since updated weights and/or an updated speechrecognition model was last provided. The one or more conditions canadditionally or alternatively include, for example, a measuredimprovement to the updated speech recognition model and/or passage of athreshold duration of time since updated weights and/or an updatedspeech recognition model was last provided. When the updated weights areprovided to the client device 110, the client device 110 can replacelocal weights, of the on-device TTS generator model 152A1, with theupdated weights. When the updated global TTS generator model is providedto the client device 110, the client device 110 can replace theon-device TTS generator model 152A1 with the updated global TTSgenerator model 152B.

Turning now to FIG. 2 , the client device 110 is illustrated in animplementation where the on-device TTS generator 122 and the on-deviceASR engine 124 of FIGS. 1A and 1B are included as part of (or incommunication with) an automated assistant client 140. The on-device TTSgenerator model 152A1 is illustrated interfacing with the on-device TTSgenerator 122, and the on-device ASR model 154A is illustratedinterfacing with the on-device ASR engine 124. Other components fromFIG. 1A are not illustrated in FIG. 2 for simplicity. FIG. 2 illustratesone example of how the on-device TTS generator 122 and on-device TTSgenerator model 152A1 can be utilized in generating synthesized speechaudio data the automated assistant client 140 causes to be rendered atthe client device 110. Further, FIG. 2 illustrates one example of howthe on-device ASR engine 124 and on-device ASR model 154A can beutilized in generating recognized text that is utilized by an automatedassistant client 140 in performing various actions.

The client device 110 in FIG. 2 is illustrated with microphone(s) 111,speaker(s) 112, camera(s) and/or other vision component(s) 113, anddisplay(s) 114 (e.g., a touch-sensitive display). The client device 110at least selectively executes the automated assistant client 140. Theautomated assistant client 140 includes, in the example of FIG. 1B, theon-device TTS generator 122, the on-device ASR engine 124, an on-devicenatural language understanding (NLU) engine 144, and an on-devicefulfillment engine 145. The automated assistant client 140 furtherincludes speech capture engine 141 and visual capture engine 142. Theautomated assistant client 140 can include additional and/or alternativeengines, such as a voice activity detector (VAD), an endpoint detector,a hotword detector, and/or other engine(s).

One or more cloud-based automated assistant components 180 canoptionally be implemented on one or more computing systems (collectivelyreferred to as a “cloud” computing system) that are communicativelycoupled to client device 110 via one or more local and/or wide areanetworks (e.g., the Internet) indicated generally at 190. Thecloud-based automated assistant components 180 can be implemented, forexample, via a cluster of high-performance servers.

In various implementations, an instance of an automated assistant client140, by way of its interactions with one or more cloud-based automatedassistant components 180, may form what appears to be, from a user'sperspective, a logical instance of an automated assistant 195 with whichthe user may engage in a human-to-computer interactions (e.g., spokeninteractions, gesture-based interactions, and/or touch-basedinteractions).

The client device 110 can be, for example: a desktop computing device, alaptop computing device, a tablet computing device, a mobile phonecomputing device, a computing device of a vehicle of the user (e.g., anin-vehicle communications system, an in-vehicle entertainment system, anin-vehicle navigation system), a standalone interactive speaker, a smartappliance such as a smart television (or a standard television equippedwith a networked dongle with automated assistant capabilities), and/or awearable apparatus of the user that includes a computing device (e.g., awatch of the user having a computing device, glasses of the user havinga computing device, a virtual or augmented reality computing device).Additional and/or alternative client devices may be provided.

The vision component(s) 113 can take various forms, such as monographiccameras, stereographic cameras, a LIDAR component (or other laser-basedcomponent(s)), a radar component, etc. The vision component(s) 113 maybe used, e.g., by visual capture engine 142, to capture vision frames(e.g., image frames, laser-based vision frames) of an environment inwhich client device 110 is deployed. In some implementations, suchvision frame(s) can be utilized to determine whether a user is presentnear the client device 110 and/or a distance of the user (e.g., theuser's face) relative to the client device. Such determination(s) can beutilized, for example, in determining whether to activate on-device ASRengine 124.

Speech capture engine 141 can be configured to capture a user's speechand/or other audio data captured via microphone(s) 111. As describedherein, such audio data can be utilized (optionally afterpre-processing) by on-device ASR engine 124. For example, on-device ASRengine 124 can process audio data that captures a spoken utterance,utilizing on-device ASR model 154A, to generate recognized text thatcorresponds to the spoken utterance. On-device NLU engine 140 performson-device natural language understanding on the recognized text togenerate NLU data. NLU engine 144 can optionally utilize one or moreon-device NLU models (not illustrated in FIG. 1B for simplicity) ingenerating the NLU data. NLU data can include, for example, intent(s)that correspond to the spoken utterance and optionally parameter(s)(e.g., slot values) for the intent(s). Further, on-device fulfillmentengine 145 generates fulfillment data using the NLU data. On-devicefulfillment engine 145 can optionally utilize one or more on-devicefulfillment models (not illustrated in FIG. 2 for simplicity) ingenerating the fulfillment data. This fulfillment data can define localand/or remote responses (e.g., answers) to the spoken utterance,interaction(s) to perform with locally installed application(s) based onthe spoken utterance, command(s) to transmit to Internet-of-things (IoT)device(s) (directly or via corresponding remote system(s)) based on thespoken utterance, and/or other resolution action(s) to perform based onthe spoken utterance. The fulfillment data is then provided for localand/or remote performance/execution of the determined action(s) toresolve the spoken utterance. Execution can include, for example,rendering local and/or remote responses (e.g., visually and/or audiblyrendering (via on-device TTS generator 122 using on-device TTS generatormodel(s) 125A1)), interacting with locally installed applications,transmitting command(s) to IoT device(s), and/or other action(s).

Display(s) 114 can be utilized to render streaming text transcriptionsfrom the on-device speech recognizer 124. Display(s) 114 can further beone of the user interface output component(s) through which visualportion(s) of a response, from automated assistant client 140, isrendered.

In some implementations, cloud-based automated assistant component(s)180 can include a remote ASR engine 182 that performs speech recognitionusing global ASR model(s) 154B, a remote NLU engine 183 that performsnatural language understanding, a remote fulfillment engine 184 thatgenerates fulfillment data, and/or a remote TTS generator 185 thatgenerates synthesized speech audio data. A remote execution module canalso optionally be included that performs remote execution based onlocal or remotely determined fulfillment data. Additional and/oralternative remote engines can be included. As described herein, invarious implementations on-device speech processing, on-device NLU,on-device fulfillment, and/or on-device execution can be prioritized atleast due to the latency and/or network usage reductions they providewhen resolving a spoken utterance (due to no client-server roundtrip(s)being needed to resolve the spoken utterance). However, one or morecloud-based automated assistant component(s) 180 can be utilized atleast selectively. For example, such component(s) can be utilized inparallel with on-device component(s) and output from such component(s)utilized when local component(s) fail. For example, on-devicefulfillment engine 145 can fail in certain situations (e.g., due torelatively limited resources of client device 160) and remotefulfillment engine 184 can utilize the more robust resources of thecloud to generate fulfillment data in such situations. Remotefulfillment engine 184 can be operated in parallel with on-devicefulfillment engine 145 and its results utilized when on-devicefulfillment fails, or can be invoked responsive to determining failureof on-device fulfillment engine 145.

In various implementations, an NLU engine (on-device 144 and/or remote183) can generate annotated output that includes one or more annotationsof the recognized text and one or more (e.g., all) of the terms of thenatural language input. In some implementations an NLU engine isconfigured to identify and annotate various types of grammaticalinformation in natural language input. For example, an NLU engine mayinclude a morphological module that may separate individual words intomorphemes and/or annotate the morphemes, e.g., with their classes. AnNLU engine may also include a part of speech tagger configured toannotate terms with their grammatical roles. Also, for example, in someimplementations an NLU engine may additionally and/or alternativelyinclude a dependency parser configured to determine syntacticrelationships between terms in natural language input.

In some implementations, an NLU engine may additionally and/oralternatively include an entity tagger configured to annotate entityreferences in one or more segments such as references to people(including, for instance, literary characters, celebrities, publicfigures, etc.), organizations, locations (real and imaginary), and soforth. In some implementations, an NLU engine may additionally and/oralternatively include a coreference resolver (not depicted) configuredto group, or “cluster,” references to the same entity based on one ormore contextual cues. In some implementations, one or more components ofan NLU engine may rely on annotations from one or more other componentsof the NLU engine.

An NLU engine may also include an intent matcher that is configured todetermine an intent of a user engaged in an interaction with automatedassistant 195. An intent matcher can use various techniques to determinean intent of the user. In some implementations, an intent matcher mayhave access to one or more local and/or remote data structures thatinclude, for instance, a plurality of mappings between grammars andresponsive intents. For example, the grammars included in the mappingscan be selected and/or learned over time, and may represent commonintents of users. For example, one grammar, “play <artist>”, may bemapped to an intent that invokes a responsive action that causes musicby the <artist> to be played on the client device 110. Another grammar,“[weather|forecast] today,” may be match-able to user queries such as“what's the weather today” and “what's the forecast for today?” Inaddition to or instead of grammars, in some implementations, an intentmatcher can employ one or more trained machine learning models, alone orin combination with one or more grammars. These trained machine learningmodels can be trained to identify intents, e.g., by embedding recognizedtext from a spoken utterance into a reduced dimensionality space, andthen determining which other embeddings (and therefore, intents) aremost proximate, e.g., using techniques such as Euclidean distance,cosine similarity, etc. As seen in the “play <artist>” example grammarabove, some grammars have slots (e.g., <artist>) that can be filled withslot values (or “parameters”). Slot values may be determined in variousways. Often users will provide the slot values proactively. For example,for a grammar “Order me a <topping> pizza,” a user may likely speak thephrase “order me a sausage pizza,” in which case the slot <topping> isfilled automatically. Other slot value(s) can be inferred based on, forexample, user location, currently rendered content, user preferences,and/or other cue(s).

A fulfillment engine (local 145 and/or remote 184) can be configured toreceive the predicted/estimated intent that is output by an NLU engine,as well as any associated slot values and fulfill (or “resolve”) theintent. In various implementations, fulfillment (or “resolution”) of theuser's intent may cause various fulfillment information (also referredto as fulfillment data) to be generated/obtained, e.g., by thefulfillment engine. This can include determining local and/or remoteresponses (e.g., answers) to the spoken utterance, interaction(s) withlocally installed application(s) to perform based on the spokenutterance, command(s) to transmit to Internet-of-things (IoT) device(s)(directly or via corresponding remote system(s)) based on the spokenutterance, and/or other resolution action(s) to perform based on thespoken utterance. The on-device fulfillment can then initiate localand/or remote performance/execution of the determined action(s) toresolve the spoken utterance.

Turning now to FIG. 3 , a flowchart illustrating an example method 300of training an on-device TTS generator model stored locally at a clientdevice is depicted. For convenience, the operations of the method 300are described with reference to a system that performs the operations.This system of method 300 includes one or more processors and/or othercomponent(s) of a client device. Moreover, while operations of themethod 300 are shown in a particular order, this is not meant to belimiting. One or more operations may be reordered, omitted, or added.

At block 352, the system determines whether one or more conditions aresatisfied. Although illustrated prior to block 354, it is noted thatblock 352 can also be performed before each of blocks 356, 358, 360,362, 364 (if included), 368 (if included), and/or 368—and/or can insteadbe performed before only a subset of blocks 356, 358, 360, 362, 364 (ifincluded), 368 (if included), and/or 368. In some implementations, block352 includes determining whether a current state of the client devicesatisfies the one or more conditions. For example, the system candetermine the current state of the client device based on sensor datafrom sensor(s) of the client device, and determine whether that currentstate of the client device satisfies the condition(s). The condition(s)can include, for example, that the client device is charging, that theclient device has at least a threshold state of charge, that the clientdevice is not currently moving and/or has not moved within a thresholdamount of time (e.g., based on sensor data from accelerometer(s),magnetometer(s), and/or other sensor(s), and/or that the client deviceis connected to an unmetered network (e.g., WiFi) such that the user ofthe client device is not charged for the receiving and/or transmittingof data. If the system determines one or more of the conditions aresatisfied, the system may proceed to block 354.

At block 354, the system identifies a given training instance, fromamong a plurality of training instances stored locally at the clientdevice of a user. Each of the plurality of training instances caninclude training instance input and training instance output. Thetraining instance input can include, for example, a given textualsegment stored locally at the client device. The training instanceoutput can include, for example, ground truth audio data correspondingto a spoken utterance of the user of the client device that correspondsto the given textual segment. The plurality of training instances can begenerated locally at the client device of the user based on spokenutterances of the user that are received at the client device. In someimplementations, in response to receiving a spoken utterance from theuser of the client device, the client device may prompt the user toverify a transcription of the spoken utterance prior to utilizing audiodata that captures the spoken utterance and/or a textual segment fromthe transcription as a training instance. In various implementations,the training examples can be generated by the device to includesemantically diverse training instances (e.g., as described above withrespect to textual segment generator 120A of FIGS. 1A and 1B).

At block 356, the system identifies a given textual segment associatedwith training instance input of the given training instance.

At block 358, the system processes, using an on-device TTS generatormodel stored locally at the client device, the given textual segment togenerate predicted synthesized speech audio data. For example, thesystem can process a sequence of phonemes corresponding to the giventextual segment, using the on-device TTS generator model, to generatethe synthesized speech audio data. In some implementations, the systemgenerates the synthesized speech audio data based on the given textualsegment along with one or more additional textual segments appendedbefore or after the given textual segment.

At block 360, the system processes, using an on-device TTS discriminatormodel stored locally at the client device, the predicted synthesizedspeech audio data to predict whether the predicted synthesized speechaudio data corresponds to: (1) a spoken utterance of the user of theclient device; or (2) synthesized speech generated by the on-device TTSgenerator model. The on-device TTS generator model utilized at block 358and the on-device TTS discriminator model utilized at block 360 may beportions of a GAN model. The goal of the on-device TTS generator modelis to generate synthesized speech audio data that includes synthesizedspeech that the one-device TTS discriminator model predicts ascorresponding to a spoken utterance of the user of the client device.

At block 362, the system generates, based on processing the predictedsynthesized speech audio data using the on-device TTS discriminatormodel, a loss. The loss may be considered in adversarial loss generatedin training the on-device TTS generator model of the GAN model. If theon-device TTS discriminator model predicts that the synthesized speechaudio data corresponds to synthesized speech generated by the on-deviceTTS generator model, then the adversarial loss may be larger than if theon-device TTS discriminator model predicts that the synthesized speechaudio data corresponds to a spoken utterance of the user of the clientdevice.

In some implementations, the method 300 of FIG. 3 may include blocks 364and 362. If blocks 364 and 366 are included, the system compares thepredicted synthesized speech audio data to ground truth audio dataassociated with training instance output of the given training example,and generates an additional loss based on the comparing at block 364.The system can compare acoustic features of the synthesized speech audiodata to the audio data of the training instance output to generate theadditional loss. The acoustic features can include, for example, audiowaveforms, mel-frequency cepstral coefficients (MFCCs), mel-filterbankfeatures, values associated with one or more prosodic properties, neuralrepresentations of the audio data (e.g., respective embeddings of theground truth audio data and the predicted synthesized speech audiodata), and/or other acoustic features of the synthesized speech audiodata and the ground truth audio data that can be compared.

At block 368, the system updates the on-device TTS generator model basedon the loss and/or the additional loss. By updating the on-device TTSgenerator model based on the loss and/or the additional loss, theon-device TTS generator model is trained to include voicecharacteristics of the user of the client device. Although the on-deviceTTS generator model is described herein as being trained to includevoice characteristics of the user, it should be understood that thesynthesized speech generated using a trained on-device TTS generatormodel is not an exact match of the speech of the user. In other words,the on-device TTS generator model can be trained to include prosodicproperties (e.g., one or more of intonation, tone, stress, rhythm,tempo, or pause) that are indicative of speech of the user of the clientdevice.

Turning now to FIG. 4 , a flowchart illustrating an example method 400of generating a gradient, locally at a client device, and utilizing thegenerated gradient to update weight(s) of an on-device ASR model and/ortransmitting the gradient to a remote system is depicted. Forconvenience, the operations of the method 400 are described withreference to a system that performs the operations. This system ofmethod 400 includes one or more processors and/or other component(s) ofa client device. Moreover, while operations of the method 400 are shownin a particular order, this is not meant to be limiting. One or moreoperations may be reordered, omitted, or added.

At block 452, the system determines whether one or more conditions aresatisfied. Although illustrated prior to block 454, it is noted thatblock 452 can also be performed before each of blocks 456, 458, 460,and/or 462—and/or can instead be performed before only a subset ofblocks 456, 458, 460, and/or 462. In some implementations, block 352includes determining whether a current state of the client devicesatisfies the one or more conditions. The one or more conditions aredescribed in greater detail herein (e.g., with respect to block 352 ofFIG. 3 ). If the system determines one or more of the conditions aresatisfied, the system may proceed to block 354.

At block 454, the system identifies a given textual segment storedlocally at a given client device of a user.

At block 456, the system processes, using a trained on-device TTSgenerator model stored locally at the client device, the given textualsegment to generate synthesized speech audio data that includessynthesized speech corresponding to the given textual segment. Thetrained on-device TTS generator model may be trained based on performingmultiple instances of the method 300 of FIG. 3 . For example, the systemcan process a sequence of phonemes corresponding to the given textualsegment, using the on-device TTS generator model, to generate thesynthesized speech audio data. In some implementations, the systemgenerates the synthesized speech audio data based on the given textualsegment along with one or more additional textual segments appendedbefore or after the given textual segment.

At block 458, the system processes, using an on-device ASR model storedlocally at the client device, the synthesized speech audio data togenerate a corresponding predicted ASR output. The correspondingpredicted ASR output can include, for example, corresponding predictedtextual segment(s), corresponding sequence(s) of predicted phoneme(s),and/or other predicted ASR output(s). In some implementations, theon-device ASR model is an end-to-end speech recognition model and thesystem generates predicted output that is the corresponding predictedtextual segment. In some other implementations, the on-device ASR modelis not an end-to-end speech recognition model, and the system generatesa sequence of predicted phonemes and/or another predictedrepresentation. The corresponding predicted textual segment can bedetermined based on the predicted phonemes and/or another predictedrepresentation.

At block 460, the system generates a gradient based on comparing thecorresponding predicted ASR output to ground truth output correspondingto the given textual segment. For example, when the system generatespredicted output that is the corresponding predicted textual segment,the predicted textual segment can be compared with the given textualsegment in generating the gradient. Also, for example, when the systemgenerates a sequence of predicted phonemes and determines the predictedtextual segment based on the sequence of predicted phonemes, thesequence of predicted phonemes can be compared with a sequence ofphonemes, that corresponds to the given textual segment, in generatingthe gradient.

At block 462, the system updates local weight(s) of the on-device ASRmodel based on the gradient of block 560 and/or transmits (e.g., via theInternet or other wide area network) the gradient of block 460 to aremote system (without transmitting any of the given textual segment,the synthesized speech audio data, and the corresponding predictedtextual segment). When the gradient is transmitted to the remote system,the remote system utilizes the generated gradient, and additionalgradients from additional client devices, to update global weight(s) ofa global ASR model. After block 462, the system then proceeds back toblock 452.

Turning now to FIG. 5 , a flowchart illustrating an example method 500of generating a gradient, locally at a client device, and utilizing thegenerated gradient to update weight(s) of an on-device TTS generatormodel and/or transmitting the gradient to a remote system is depicted.For convenience, the operations of the method 500 are described withreference to a system that performs the operations. This system ofmethod 500 includes one or more processors and/or other component(s) ofa client device. Moreover, while operations of the method 500 are shownin a particular order, this is not meant to be limiting. One or moreoperations may be reordered, omitted, or added.

At block 552, the system determines whether one or more conditions aresatisfied. Although illustrated prior to block 454, it is noted thatblock 552 can also be performed before each of blocks 556, 558, 560,562, and/or 564—and/or can instead be performed before only a subset ofblocks 556, 558, 560, 562, and/or 564. In some implementations, block552 includes determining whether a current state of the client devicesatisfies the one or more conditions. The one or more conditions aredescribed in greater detail herein (e.g., with respect to block 352 ofFIG. 3 ). If the system determines one or more of the conditions aresatisfied, the system may proceed to block 554.

At block 554, the system identifies a given textual segment storedlocally at a given client device of a user.

At block 556, the system processes, using a trained on-device TTSgenerator model stored locally at the client device, the given textualsegment to generate synthesized speech audio data that includessynthesized speech corresponding to the given textual segment. Forexample, the system can process a sequence of phonemes corresponding tothe given textual segment, using the on-device TTS generator model, togenerate the synthesized speech audio data. In some implementations, thesystem generates the synthesized speech audio data based on the giventextual segment along with one or more additional textual segmentsappended before or after the given textual segment.

At block 558, the system processes, using a trained on-device TTSdiscriminator model stored locally at the client device, the synthesizedspeech audio data to predict whether the synthesized speech audio datacorresponds to: (1) a spoken utterance of the user of the client device;or (2) synthesized speech generated by the on-device TTS generatormodel. The on-device TTS generator model utilized at block 556 and theon-device TTS discriminator model utilized at block 558 may be portionsof a GAN model. The goal of the on-device TTS generator model is togenerate synthesized speech audio data that includes synthesized speechthat the one-device TTS discriminator model predicts as corresponding toa spoken utterance of the user of the client device.

In some implementations, the method 500 may include optional block 560.If block 560 is included, the system identifies ground truth audio datacorresponding to a spoken utterance of the user, the textual segmentbeing a ground truth textual segment for the ground truth audio data.The ground truth audio data can be stored in association with the giventextual segment stored locally at the given client device (e.g., inon-device storage 111 of FIGS. 1A and 1B). In some versions of thoseimplementations, block 560 may include optional sub-block 560A. Ifsub-block 560A is included, the system filters the ground truth audiodata to remove additional spoken utterances and/or ambient noise. Thesystem can filter the ground truth audio data to remove the additionalspoken utterances by using one or more known techniques. For example,the system can utilize a voice filtering model that process the groundtruth audio data and a speaker embedding associated with the user of theclient device to generate ground truth audio data that only includes thespoken utterance of the user and removes any audio data that does notmatch the speaker embedding associated with the user of the clientdevice. As another example, the system can utilize a filtering model oralgorithm to remove the ambient noise to generate ground truth audiodata that isolates the spoken utterance of the user.

At block 562, the system generates a gradient based on the processing bythe trained on-device TTS discriminator model at block 558. The gradientgenerated at block 562 may be similar to the adversarial loss for theGAN model (e.g., described with respect to block 362 of FIG. 3 ). Insome implementations when optional block 560 is included, block 562 mayinclude sub-block 562A. If sub-block 562A is included, the systemgenerates a gradient based on comparing the synthesized speech audiodata to the ground truth audio data. The system can compare acousticfeatures of the synthesized speech audio data to the ground truth audiodata to generate the gradient. The acoustic features can include, forexample, audio waveforms, mel-frequency cepstral coefficients (MFCCs),mel filterbank features, values associated with one or more prosodicproperties, and/or other acoustic features of the synthesized speechaudio data and the ground truth audio data that can be compared.

At block 564, the system updates local weight(s) of the on-device TTSgenerator model based on the gradient of block 562 (and optionally theadditional gradient of sub-block 562A) and/or transmits (e.g., via theInternet or other wide area network) the gradient of block 562 (andoptionally the additional gradient of sub-block 562A) to a remote system(without transmitting any of the given textual segment, and thesynthesized speech audio data). When the gradient(s) are transmitted tothe remote system, the remote system utilizes the generated gradient,and additional gradients from additional client devices, to updateglobal weight(s) of a global TTS generator model. After block 564, thesystem then proceeds back to block 552.

Turning now to FIG. 6 a flowchart illustrating an example method 600 oftransmitting, to a remote system, gradient(s) generated locally at aclient device that is performing corresponding instances of the method400 of FIG. 4 and/or the method 500 of FIG. 5 is depicted. Forconvenience, the operations of the method 600 are described withreference to a system that performs the operations. This system ofmethod 600 includes one or more processors and/or other component(s) ofa client device. Moreover, while operations of the method 600 are shownin a particular order, this is not meant to be limiting. One or moreoperations may be reordered, omitted, or added.

At block 652, the system determines whether one or more conditions aresatisfied. Although illustrated prior to block 654, it is noted thatblock 652 can also be performed before block 656. In someimplementations, block 652 includes determining whether a current stateof the client device satisfies the one or more conditions. The one ormore conditions are described in greater detail herein (e.g., withrespect to block 352 of FIG. 3 ). If the system determines one or moreof the conditions are satisfied, the system may proceed to block 654.

At block 654, the system determines whether a client device hasgenerated gradient(s) by performing corresponding instances of themethod 400 of FIG. 4 and/or the method 500 of FIG. 5 . If the systemdetermines that no gradient(s) have been generated at the client device,the system can continuously monitor whether any gradient(s) aregenerated at the client device while the condition(s) are satisfied atblock 652. If the system determines that gradient(s) have been generatedat the client device, then the system may proceed to block 656.

In some implementations, the method 600 may optionally include blocks656 and 658. If block 656 is included, the system may determine whetherany audio data (e.g., ground truth audio data) utilized in generatingany of the gradient(s) were generated based on audio data that includesportion(s) of additional spoken utterances of additional humans that arein addition to the user of the client device and/or ambient noise thatsatisfies a noise threshold (e.g., as described with respect to optionalsub-block 558A of FIG. 5 ). If the system determines that thegradient(s) were generated based on audio data that includes theportion(s) of the additional spoken utterances and/or the ambient noisethat satisfies the noise threshold, but the audio data was not filtered,then the system may proceed to block 658. At block 658, the systemrefrains from transmitting the gradient(s) to a remote system. Althoughthe system may refrain from transmitting the gradient(s) to the remotesystem, the system may still update an on-device model (e.g., anon-device TTS generator model and/or an on-device ASR model) based onthe gradient(s). However, if the audio data was filtered, then thesystem may proceed to block 660. In implementations that omit blocks 656and block 660, the system may proceed directly from block 654 to block660.

At block 660, the system transmits (e.g., via the Internet or other widearea network) the gradient(s) of block 654 to a remote system (withouttransmitting data on which the gradient(s) are generated). When thegradient(s) are transmitted to the remote system, the remote systemutilizes the generated gradient(s), and additional gradients fromadditional client devices, to update global weight(s) of globalmodel(s). For example, the system can update weight(s) of a global ASRmodel if the gradient(s) of block 654 include gradient(s) generatedbased on instances of the method 400 of FIG. 4 . Further, the system canupdate weight(s) of a global TTS generator model if the gradient(s) ofblock 654 include gradient(s) generated based on instances of the method500 of FIG. 5 . After block 660, the system then proceeds back to block652.

Turning now to FIG. 7 , a flowchart illustrating an example method 700of updating weight(s) of global model(s) based on gradient(s) receivedfrom remote client device(s) that are performing corresponding instancesof method 400 of FIG. 4 and/or method 500 of FIG. 5 , and transmitting,to remote client device(s), the updated weight(s) and/or the updatedglobal model(s) is depicted. For convenience, the operations of themethod 300 are described with reference to a system that performs theoperations. This system may include various components of variouscomputer systems, such as one or more server devices. Moreover, whileoperations of the method 700 are shown in a particular order, this isnot meant to be limiting. One or more operations may be reordered,omitted, or added.

At block 752, the system receives gradient(s) from remote clientdevice(s). For example, the system can receive gradient(s) from aplurality of remote client devices that are performing correspondinginstances of the method 400 of FIG. 4 and/or the method 500 of FIG. 5 .

At block 754, the system updates weight(s) of a global model based onthe gradient(s) received at block 752. In some implementations, theglobal model may be a global ASR model. The system can update theweight(s) of the global ASR model based on any gradient(s) received atblock 752 that are generated based on the instances of the method 400 ofFIG. 4 . In some additional and/or alternative implementations, theglobal model may be a global TTS generator model. The system can updatethe weight(s) of the global TTS model based on any gradient(s) receivedat block 752 that are generated based on the instances of the method 600of FIG. 6 . Iterations of blocks 752 and 754 can continue to beperformed as new gradient(s) are received and/or are queued after beingreceived.

At block 756, the system at least periodically determines whether one ormore conditions are satisfied, such as one or more of the conditionsdescribed herein (e.g., with respect to the update distribution engine164 of FIGS. 1A and 1B). Generally, the condition(s) serve as a proxyfor determining whether the global model has been updated to an extentthat justifies utilization of network resources in transmitting updatedweight(s) for the global model and/or the updated global model itself.In other words, the condition(s) are used as a proxy for determining ifthe performance gains of the model justify the usage of networkresources. If so, the system proceeds to block 758 and transmits, to aplurality of client devices, current updated global weight(s) and/or acurrent updated global model itself. The updated weight(s) and/or theupdated global model can optionally be transmitted to a given clientdevice responsive to a request from the given client device, such as arequest during an update procedure and/or a request sent due to theclient device being idle and/or charging.

Turning now to FIG. 8 , a block diagram of an example computing device810 that may optionally be utilized to perform one or more aspects oftechniques described herein is depicted. In some implementations, one ormore of a client device, cloud-based automated assistant component(s),and/or other component(s) may comprise one or more components of theexample computing device 810.

Computing device 810 typically includes at least one processor 814 whichcommunicates with a number of peripheral devices via bus subsystem 812.These peripheral devices may include a storage subsystem 824, including,for example, a memory subsystem 825 and a file storage subsystem 826,user interface output devices 820, user interface input devices 822, anda network interface subsystem 816. The input and output devices allowuser interaction with computing device 810. Network interface subsystem816 provides an interface to outside networks and is coupled tocorresponding interface devices in other computing devices.

User interface input devices 822 may include a keyboard, pointingdevices such as a mouse, trackball, touchpad, or graphics tablet, ascanner, a touchscreen incorporated into the display, audio inputdevices such as voice recognition systems, microphones, and/or othertypes of input devices. In general, use of the term “input device” isintended to include all possible types of devices and ways to inputinformation into computing device 810 or onto a communication network.

User interface output devices 820 may include a display subsystem, aprinter, a fax machine, or non-visual displays such as audio outputdevices. The display subsystem may include a cathode ray tube (CRT), aflat-panel device such as a liquid crystal display (LCD), a projectiondevice, or some other mechanism for creating a visible image. Thedisplay subsystem may also provide non-visual display such as via audiooutput devices. In general, use of the term “output device” is intendedto include all possible types of devices and ways to output informationfrom computing device 810 to the user or to another machine or computingdevice.

Storage subsystem 824 stores programming and data constructs thatprovide the functionality of some or all of the modules describedherein. For example, the storage subsystem 824 may include the logic toperform selected aspects of the methods disclosed herein, as well as toimplement various components depicted in FIGS. 1A, 1B, and 2 .

These software modules are generally executed by processor 814 alone orin combination with other processors. Memory 825 used in the storagesubsystem 824 can include a number of memories including a main randomaccess memory (RAM) 830 for storage of instructions and data duringprogram execution and a read only memory (ROM) 832 in which fixedinstructions are stored. A file storage subsystem 826 can providepersistent storage for program and data files, and may include a harddisk drive, a floppy disk drive along with associated removable media, aCD-ROM drive, an optical drive, or removable media cartridges. Themodules implementing the functionality of certain implementations may bestored by file storage subsystem 826 in the storage subsystem 824, or inother machines accessible by the processor(s) 814.

Bus subsystem 812 provides a mechanism for letting the variouscomponents and subsystems of computing device 810 communicate with eachother as intended. Although bus subsystem 812 is shown schematically asa single bus, alternative implementations of the bus subsystem may usemultiple busses.

Computing device 810 can be of varying types including a workstation,server, computing cluster, blade server, server farm, or any other dataprocessing system or computing device. Due to the ever-changing natureof computers and networks, the description of computing device 810depicted in FIG. 8 is intended only as a specific example for purposesof illustrating some implementations. Many other configurations ofcomputing device 810 are possible having more or fewer components thanthe computing device depicted in FIG. 8 .

In situations in which the systems described herein collect or otherwisemonitor personal information about users, or may make use of personaland/or monitored information), the users may be provided with anopportunity to control whether programs or features collect userinformation (e.g., information about a user's social network, socialactions or activities, profession, a user's preferences, or a user'scurrent geographic location), or to control whether and/or how toreceive content from the content server that may be more relevant to theuser. Also, certain data may be treated in one or more ways before it isstored or used, so that personal identifiable information is removed.For example, a user's identity may be treated so that no personalidentifiable information can be determined for the user, or a user'sgeographic location may be generalized where geographic locationinformation is obtained (such as to a city, ZIP code, or state level),so that a particular geographic location of a user cannot be determined.Thus, the user may have control over how information is collected aboutthe user and/or used.

In some implementations, a method implemented by one or more processorsis provided, and includes training, based on a plurality of traininginstances, an on-device text-to-speech (TTS) generator model. Theon-device TTS generator model is a portion of a generative adversarialnetwork (GAN) model stored locally at the client device, where the GANmodel also includes an on-device TTS discriminator model. Training theTTS generator model includes training the on-device TTS generator modelto generate synthesized speech audio data that includes voicecharacteristics of the user of the client device. The method furtherincludes, subsequent to training the on-device TTS generator model,identifying a textual segment stored locally at the client device of auser, processing, using the trained on-device TTS generator model storedlocally at the client device, the textual segment to generate additionalsynthesized speech audio data that includes synthesized speechcorresponding to the textual segment, processing, using an on-deviceautomatic speech recognition (ASR) model stored locally at the clientdevice, the additional synthesized speech audio data to generate acorresponding predicted ASR output, generating a gradient based oncomparing the corresponding predicted ASR output to ground truth outputcorresponding to the textual segment, and updating local weights of theon-device ASR model based on the generated gradient.

These and other implementations of technology disclosed herein canoptionally include one or more of the following features.

In some implementations, the plurality of training instances may begenerator training instances, and each of the plurality of traininginstances include training instance input and training instance output.The training instance input may include a given textual segment storedlocally at the client device, and the training instance output mayinclude a ground truth label. Training the on-device TTS generator modelstored locally at the client device based on a given training instance,of the plurality of training instances, may include processing, usingthe on-device TTS generator model, the given textual segment to generatepredicted synthesized speech audio data that includes predictedsynthesized speech corresponding to the given textual segment,processing, using the on-device TTS discriminator model, the predictedsynthesized speech audio data to predict whether it corresponds to anactual spoken utterance of the user of the client device or thepredicted synthesized speech generated by the on-device TTS generatormodel, and generating, based on the ground truth label and based onprocessing using the on-device TTS discriminator model, a loss. In someversions of those implementations, the training instance output mayfurther include ground truth audio data that includes a given spokenutterance of the user that corresponds to the given textual segment.Training the on-device TTS generator model stored locally at the clientdevice based on the given training instance, of the plurality oftraining instances, may further include comparing the predictedsynthesized speech audio data that includes the predicted synthesizedspeech to the ground truth audio data that includes the given spokenutterance of the user, and generating, based on comparing the predictedsynthesized speech audio data to the ground truth audio data, anadditional loss. In some additional or alternative versions of thoseimplementations, comparing the predicted synthesized speech audio datato the ground truth audio data includes comparing acoustic features ofthe ground truth audio data to synthesized acoustic features of thepredicted synthesized speech audio data. In some additional oralternative versions of those implementations, the method may furtherinclude updating the on-device TTS generator model based one or more ofthe loss or the additional loss. Updating the on-device TTS generatormodel based one or more of the loss or the additional loss may includebackpropagating one or more of the loss or the additional loss acrossthe on-device TTS generator model.

In some implementations, the method may further include, prior totraining the on-device TTS generator model, training, based on aplurality of additional training instances, the on-device TTSdiscriminator model. In some versions of those implementations, theplurality of additional training instances may be discriminator traininginstances, and each of the plurality of additional training instancesmay include additional training instance input and additional traininginstance output. The additional training instance input may includegiven audio data that includes a given spoken utterance of the user ofthe client device or synthesized speech audio data that includessynthesized speech generated by the on-device TTS generator model, andthe training instance output may include a ground truth label thatindicates whether the additional training instance input corresponds tothe given audio data or the synthesized speech audio data. Training theon-device TTS discriminator model stored locally at the client devicebased on a given training instance, of the plurality of additionaltraining instances, may include processing, using the on-device TTSdiscriminator model, the given training instance input to predictwhether it corresponds to an actual spoken utterance of the user of theclient device or the synthesized speech generated by the on-device TTSgenerator model, generating, based on the ground truth label and basedon the processing using the on-device TTS discriminator model, a loss,and updating the on-device TTS discriminator model based on the loss.

In some implementations, the voice characteristics of the user of theclient device may include prosodic properties of a voice of the user,where the prosodic properties of the voice of the user comprise one ormore of: intonation, tone, stress, rhythm, tempo, and pause.

In some implementations, the method may further include transmitting,over a network and to a remote system, the generated gradient to theremote system without transmitting any of: the additional textualsegment, the additional synthesized speech audio data, and thecorresponding predicted ASR output. The remote system may utilize thegenerated gradient, and additional gradients from additional clientdevices, to update global weights of a global ASR model. In someversions of those implementations, the method may further includereceiving, at the client device and from the remote system, the globalASR model or the updated global weights. Receiving the global ASR modelor the updated global weights may be subsequent to the remote systemupdating the global weights of the global ASR model based on thegradient and the additional gradients. The method may further include,responsive to receiving the global ASR model or the updated globalweights, replacing, in local storage of the client device, the on-deviceASR model with the global ASR model or the local weights of theon-device ASR model with the updated global weights of the global ASRmodel.

In some implementations, a method implemented by one or more processorsis provided, and includes identifying a textual segment stored locallyat the client device of a user, processing, using a trained on-devicetext-to-speech (US) generator model stored locally at the client device,the textual segment to generate synthesized speech audio data thatincludes synthesized speech corresponding to the textual segment,processing, using a trained on-device TTS discriminator model storedlocally at the client device, the synthesized speech audio data thatincludes synthesized speech corresponding to the textual segment todetermine whether the synthesized speech corresponds to the synthesizedspeech audio data generated by the trained on-device TTS generator modelor a spoken utterance of the user of the client device, generating agradient based on the processing by the trained on-device TTSdiscriminator model, and updating local weights of the on-device TTSmodel based on the generated gradient.

These and other implementations of technology disclosed herein canoptionally include one or more of the following features.

In some implementations, the method may further include transmitting,over a network and to a remote system, the generated gradient to theremote system without transmitting any of: the textual segment, and thesynthesized speech audio data. The remote system may utilize thegenerated gradient, and additional gradients from additional clientdevices, to update global weights of a global TTS model.

In some versions of those implementations, the method may furtherinclude identifying ground truth audio data corresponding to a givenspoken utterance of the user, the textual segment being a ground truthtextual segment for the ground truth audio data, and generating anadditional gradient based on comparing the synthesized speech audio dataand the ground truth audio data. Updating the local weights of theon-device TTS model may be further based on the generated additionalgradient. Transmitting, over the network and to the remote system, thegenerated gradient to the remote system may further include transmittingthe generated additional gradient without transmitting the ground truthaudio data. In some further versions of those implementations, themethod may further include determining whether to transmit the generatedadditional gradient to the remote system. Determining whether totransmit the generated additional gradient to the remote system mayinclude determining whether the ground truth audio data utilized ingenerating the additional gradient captures one or more of: additionalaudio data corresponding to a portion of an additional utterance of anadditional user, or ambient noise that satisfies a noise threshold. Themethod may further include, in response to determining the ground truthaudio data utilized in generating the additional gradient captures theadditional audio data corresponding to the portion of the additionalutterance of the additional user, or the ambient noise that satisfiesthe noise threshold, refraining from transmitting the additionalgenerated gradient to the remote system to update the global weights ofa global TTS model. In some additional or alternative further versionsof those implementations, the method may further include determiningwhether the ground truth audio data utilized in generating theadditional gradient captures one or more of: additional audio datacorresponding to a portion of an additional utterance of an additionaluser, or ambient noise that satisfies a noise threshold, and, inresponse to determining the ground truth audio data utilized ingenerating the additional gradient captures the additional audio datacorresponding to the portion of the additional utterance of theadditional user, or the ambient noise that satisfies the noisethreshold, and prior to generating the gradient, filtering the groundtruth audio data to remove the additional audio data corresponding tothe portion of the additional utterance of the additional user, or theambient noise.

In some additional or alternative versions of those implementations, themethod may further include receiving, at the client device and from theremote system, the updated global TTS model or the updated globalweights. Receiving the global TTS model or the updated global weights issubsequent to the remote system updating the global weights of theglobal TTS model based on the gradient and the additional gradients. Themethod may further include, responsive to receiving the global TTS modelor the updated global weights, replacing, in local storage of the clientdevice, the on-device TTS model with the global TTS model or localweights of the on-device TTS model with the updated global weights.

In some additional or alternative versions of those implementations, theglobal TTS model may be one of a plurality of disparate global TTSmodels that correspond to a plurality of distinct languages, and theglobal TTS model, of the plurality of disparate TTS models, maycorrespond to a given language, of the plurality of distinct languages,associated with the user of the client device.

In some additional or alternative versions of those implementations, theglobal TTS model may be one of a plurality of disparate global TTSmodels that correspond to a plurality of distinct geographical regions,and the global TTS model, of the plurality of disparate TTS models, maycorrespond to a given geographical region, of the plurality of distinctgeographical regions, associated with the user of the client device.

In some implementations, the method further includes generating, basedon the textual segment, a plurality of alternate textual segments thatare semantically diverse from the textual segment, processing, using theon-device TTS generator model stored locally at the client device, agiven alternate segment, of the plurality of alternate textual segments,to generate alternate synthesized speech audio data that includesalternate synthesized speech corresponding to the given alternatetextual segment, identifying additional ground truth audio datacorresponding to an alternate spoken utterance of the user of the clientdevice, the alternate textual segment being a ground truth alternatetextual segment for the additional ground truth audio data, comparingthe synthesized speech audio data to the ground truth audio data,generating an additional gradient based on comparing the alternatesynthesized speech audio data to the additional ground truth audio data,and further updating one or more the local weights of the on-device TTSgenerator model based on the generated additional gradient.

In some implementations, a method implemented by one or more processorsis provided, and includes identifying a textual segment stored locallyat the client device of a user, processing, using a trained on-devicetext-to-speech (TTS) generator model stored locally at the clientdevice, the textual segment to generate synthesized speech audio datathat includes synthesized speech corresponding to the textual segment,processing, using a trained on-device TTS discriminator model storedlocally at the client device, the synthesized speech audio data thatincludes synthesized speech corresponding to the textual segment todetermine whether the synthesized speech corresponds to the synthesizedspeech audio data generated by the trained on-device TTS generator modelor a spoken utterance of the user of the client device, generating agradient based on the processing by the trained on-device TTSdiscriminator model, and transmitting, over a network and to a remotesystem, the generated gradient to the remote system, wherein the remotesystem utilizes the generated gradient, and additional gradients fromadditional client devices, to update global weights of a global TTSmodel.

In addition, some implementations include one or more processors (e.g.,central processing unit(s) (CPU(s)), graphics processing unit(s)(GPU(s), and/or tensor processing unit(s) (TPU(s)) of one or morecomputing devices, where the one or more processors are operable toexecute instructions stored in associated memory, and where theinstructions are configured to cause performance of any of theaforementioned methods. Some implementations also include one or morenon-transitory computer readable storage media storing computerinstructions executable by one or more processors to perform any of theaforementioned methods. Some implementations also include a computerprogram product including instructions executable by one or moreprocessors to perform any of the aforementioned methods.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts described in greater detail herein arecontemplated as being part of the subject matter disclosed herein. Forexample, all combinations of claimed subject matter appearing at the endof this disclosure are contemplated as being part of the subject matterdisclosed herein.

What is claimed is:
 1. A method implemented by one or more processors, the method comprising: training, based on a plurality of training instances, an on-device text-to-speech (TTS) generator model, wherein the on-device TTS generator model is a portion of a generative adversarial network (GAN) model stored locally at a client device, wherein the GAN model also includes an on-device TTS discriminator model, and wherein training the TTS generator model comprises training the on-device TTS generator model to generate synthesized speech audio data that includes voice characteristics of the user of the client device; and subsequent to training the on-device TTS generator model: identifying a textual segment stored locally at the client device of a user; processing, using the trained on-device TTS generator model stored locally at the client device, the textual segment to generate additional synthesized speech audio data that includes synthesized speech corresponding to the textual segment; processing, using an on-device automatic speech recognition (ASR) model stored locally at the client device, the additional synthesized speech audio data to generate a corresponding predicted ASR output; generating a gradient based on comparing the corresponding predicted ASR output to ground truth output corresponding to the textual segment; and updating local weights of the on-device ASR model based on the generated gradient.
 2. The method of claim 1, wherein the plurality of training instances are generator training instances; wherein each of the plurality of training instances include training instance input and training instance output, the training instance input including a given textual segment stored locally at the client device, and the training instance output including a ground truth label; and wherein training the on-device TTS generator model stored locally at the client device based on a given training instance, of the plurality of training instances, comprises: processing, using the on-device TTS generator model, the given textual segment to generate predicted synthesized speech audio data that includes predicted synthesized speech corresponding to the given textual segment; processing, using the on-device TTS discriminator model, the predicted synthesized speech audio data to predict whether it corresponds to an actual spoken utterance of the user of the client device or the predicted synthesized speech generated by the on-device TTS generator model; and generating, based on the ground truth label and based on processing using the on-device TTS discriminator model, a loss.
 3. The method of claim 2, wherein the training instance output further includes ground truth audio data that includes a given spoken utterance of the user that corresponds to the given textual segment; and wherein training the on-device TTS generator model stored locally at the client device based on the given training instance, of the plurality of training instances, further comprises: comparing the predicted synthesized speech audio data that includes the predicted synthesized speech to the ground truth audio data that includes the given spoken utterance of the user; and generating, based on comparing the predicted synthesized speech audio data to the ground truth audio data, an additional loss.
 4. The method of claim 3, wherein comparing the predicted synthesized speech audio data to the ground truth audio data comprises: comparing acoustic features of the ground truth audio data to synthesized acoustic features of the predicted synthesized speech audio data.
 5. The method of claim 3, further comprising: updating the on-device TTS generator model based one or more of the loss or the additional loss; and wherein updating the on-device TTS generator model based one or more of the loss or the additional loss comprises backpropagating one or more of the loss or the additional loss across the on-device TTS generator model.
 6. The method of claim 1, further comprising: prior to training the on-device TTS generator model: training, based on a plurality of additional training instances, the on-device TTS discriminator model.
 7. The method of claim 6, wherein the plurality of additional training instances are discriminator training instances; wherein each of the plurality of additional training instances include additional training instance input and additional training instance output, the additional training instance input including given audio data that includes a given spoken utterance of the user of the client device or synthesized speech audio data that includes synthesized speech generated by the on-device TTS generator model, and the training instance output including a ground truth label that indicates whether the additional training instance input corresponds to the given audio data or the synthesized speech audio data; and wherein training the on-device TTS discriminator model stored locally at the client device based on a given training instance, of the plurality of additional training instances, comprises: processing, using the on-device TTS discriminator model, the given training instance input to predict whether it corresponds to an actual spoken utterance of the user of the client device or the synthesized speech generated by the on-device TTS generator model; generating, based on the ground truth label and based on the processing using the on-device TTS discriminator model, a loss; and updating the on-device TTS discriminator model based on the loss.
 8. The method of claim 1, wherein the voice characteristics of the user of the client device comprise prosodic properties of a voice of the user, wherein the prosodic properties of the voice of the user comprise one or more of: intonation, tone, stress, rhythm, tempo, and pause.
 9. The method of claim 1, further comprising: transmitting, over a network and to a remote system, the generated gradient to the remote system without transmitting any of: the additional textual segment, the additional synthesized speech audio data, and the corresponding predicted ASR output, and wherein the remote system utilizes the generated gradient, and additional gradients from additional client devices, to update global weights of a global ASR model.
 10. The method of claim 9, further comprising: receiving, at the client device and from the remote system, the global ASR model or the updated global weights, wherein receiving the global ASR model or the updated global weights is subsequent to the remote system updating the global weights of the global ASR model based on the gradient and the additional gradients; and responsive to receiving the global ASR model or the updated global weights, replacing, in local storage of the client device, the on-device ASR model with the global ASR model or the local weights of the on-device ASR model with the updated global weights of the global ASR model.
 11. A method implemented by one or more processors of a client device, the method comprising: identifying a textual segment stored locally at the client device of a user; processing, using a trained on-device text-to-speech (TTS) generator model stored locally at the client device, the textual segment to generate synthesized speech audio data that includes synthesized speech corresponding to the textual segment; processing, using a trained on-device TTS discriminator model stored locally at the client device, the synthesized speech audio data that includes synthesized speech corresponding to the textual segment to determine whether the synthesized speech corresponds to the synthesized speech audio data generated by the trained on-device TTS generator model or a spoken utterance of the user of the client device; generating a gradient based on the processing by the trained on-device TTS discriminator model; and updating local weights of the on-device TTS generator model based on the generated gradient.
 12. The method of claim 11, further comprising: transmitting, over a network and to a remote system, the generated gradient to the remote system without transmitting any of: the textual segment, and the synthesized speech audio data, wherein the remote system utilizes the generated gradient, and additional gradients from additional client devices, to update global weights of a global TTS model.
 13. The method of claim 12, further comprising: identifying ground truth audio data corresponding to a given spoken utterance of the user, the textual segment being a ground truth textual segment for the ground truth audio data; generating an additional gradient based on comparing the synthesized speech audio data and the ground truth audio data; wherein updating the local weights of the on-device TTS generator model is further based on the generated additional gradient; and wherein transmitting, over the network and to the remote system, the generated gradient to the remote system further comprising transmitting the generated additional gradient without transmitting the ground truth audio data.
 14. The method of claim 13, further comprising: determining whether to transmit the generated additional gradient to the remote system, wherein determining whether to transmit the generated additional gradient to the remote system comprises: determining whether the ground truth audio data utilized in generating the additional gradient captures one or more of: additional audio data corresponding to a portion of an additional utterance of an additional user, or ambient noise that satisfies a noise threshold; and in response to determining the ground truth audio data utilized in generating the additional gradient captures the additional audio data corresponding to the portion of the additional utterance of the additional user, or the ambient noise that satisfies the noise threshold: refraining from transmitting the additional generated gradient to the remote system to update the global weights of a global TTS model.
 15. The method of claim 13, further comprising: determining whether the ground truth audio data utilized in generating the additional gradient captures one or more of: additional audio data corresponding to a portion of an additional utterance of an additional user, or ambient noise that satisfies a noise threshold; and in response to determining the ground truth audio data utilized in generating the additional gradient captures the additional audio data corresponding to the portion of the additional utterance of the additional user, or the ambient noise that satisfies the noise threshold, and prior to generating the gradient: filtering the ground truth audio data to remove the additional audio data corresponding to the portion of the additional utterance of the additional user, or the ambient noise.
 16. The method of claim 12, further comprising: receiving, at the client device and from the remote system, the updated global TTS model or the updated global weights, wherein receiving the global TTS model or the updated global weights is subsequent to the remote system updating the global weights of the global TTS model based on the gradient and the additional gradients; and responsive to receiving the global TTS model or the updated global weights, replacing, in local storage of the client device, the on-device TTS generator model with the global TTS model or local weights of the on-device TTS generator model with the updated global weights.
 17. The method of claim 12, wherein the global TTS model is one of a plurality of disparate global TTS models that correspond to a plurality of distinct languages, and wherein the global TTS model, of the plurality of disparate TTS models, corresponds to a given language, of the plurality of distinct languages, associated with the user of the client device.
 18. The method of claim 12, wherein the global TTS model is one of a plurality of disparate global TTS models that correspond to a plurality of distinct geographical regions, and wherein the global TTS model, of the plurality of disparate TTS models, corresponds to a given geographical region, of the plurality of distinct geographical regions, associated with the user of the client device.
 19. The method of claim 11, further comprising: generating, based on the textual segment, a plurality of alternate textual segments that are semantically diverse from the textual segment. processing, using the on-device TTS generator model stored locally at the client device, a given alternate segment, of the plurality of alternate textual segments, to generate alternate synthesized speech audio data that includes alternate synthesized speech corresponding to the given alternate textual segment; identifying additional ground truth audio data corresponding to an alternate spoken utterance of the user of the client device, the alternate textual segment being a ground truth alternate textual segment for the additional ground truth audio data; comparing the synthesized speech audio data to the ground truth audio data; generating an additional gradient based on comparing the alternate synthesized speech audio data to the additional ground truth audio data; and further updating one or more the local weights of the on-device TTS generator model based on the generated additional gradient.
 20. A method implemented by one or more processors of a client device, the method comprising: identifying a textual segment stored locally at the client device of a user; processing, using a trained on-device text-to-speech (TTS) generator model stored locally at the client device, the textual segment to generate synthesized speech audio data that includes synthesized speech corresponding to the textual segment; processing, using a trained on-device TTS discriminator model stored locally at the client device, the synthesized speech audio data that includes synthesized speech corresponding to the textual segment to determine whether the synthesized speech corresponds to the synthesized speech audio data generated by the trained on-device TTS generator model or a spoken utterance of the user of the client device; generating a gradient based on the processing by the trained on-device TTS discriminator model; and transmitting, over a network and to a remote system, the generated gradient to the remote system, wherein the remote system utilizes the generated gradient, and additional gradients from additional client devices, to update global weights of a global TTS model. 